Detector and pixel corrections
Detector and pixel corrections
Some detectors are better than others. All results trace back to the detectors, and errors in the detector propagate through all results. A little effort goes a long way towards improving results.
dark calibration file
Signal on the detector with no incident photons.
Usually electronic offsets. Detector offsets can change, especially when detector systems are swapped or per-pixel gain settings are adjusted. It is good practice to use a new dark calibration for each new experiment. The weaker your signals the more important this becomes - single photon signals are only a few detector values (ADU) in size and can get easily lost when using the wrong dark calibration file. In extreme cases, more frequent darkcals may be needed.
In the .ini file:
# Dark calibration
darkcal=../../calib/darkcal/current-darkcal.h5
Configuration for producing a darkcal file is in: darkcal.ini
> less /reg/g/cfel/cheetah/template/cheetah/process/darkcal.ini
[front]
detectorType=cspad
detectorName=CxiDs1
saveDetectorRaw=0
saveDetectorCorrected=1
saveDetectorAndPhotonCorrected=0
saveAssembled=0
[]
hitfinder=0
generateDarkcal=1
powderSumHits=1
powderSumBlanks=1
saveInterval=500
nThreads=16
Note: the mean and standard deviation contained in the *-class0-sum.h5 file is very useful for finding bad pixels.
bad pixel mask
Bad pixels are flagged in the pixel mask and set to zero at the start of analysis. Bad pixels are always ignored in analysis (radial averages, peak finding, images etc). Zeroing the pixels solves problems with auto-ranging viewers displaying the image on a useful image scale (outliers throw off autoranging and these pixels are definitely bad so kill them).
In the .ini file:
# Bad pixel maps
badpixelmap=../../calib/mask/current-badpix.h5
Every detector has bad pixels. These need to be identified so that they can be avoided during analysis.
Bad pixels can be identified from dark frames as pixels that are either excessively noisy or have offsets significantly different to their neighbours. Use darkcal.ini to process a dark run, then “Tools-->Make bad pixel mask from Darkcal” to identify misbehaving pixels.
Dead pixels behave the same whether or not they are illuminated and are therefore useless for signal measurement. However they can’t be identified from the dark calibration file because they look identical to normal pixels when not illuminated. Dead pixels can be identified from runs in which photons are incident on each pixel (for example, water scattering). Use brightfield.ini to sum up one such run, then “Tools--> Make bad pixel mask from brightfield” to identify pixels which remain dark even when illuminated. There are corner cases which go wrong - the results of this need to be inspected.
Masks can be combined using the “Tools-->Combine masks” utility. Use ‘Shift’ to select the masks to be combined.
geometry
Is absolutely critical for analysis, although hit finding itself is normally not very sensitive to a perfect geometry.
Unlike crystal indexing, and Cheetah can happily run using the provided default geometry files. Indeed this is often adequate. However, radial background subtraction is sensitive to geometry, and can be quite so in the presence of concentric ‘powder rings’ from ice or other polycrystalline sample in the beam. SAXS/WAXS traces also benefit from use of a refined geometry so that radial averages are averaged across the same scattering angles.
In the .ini file:
# Detector geometry
geometry=../../calib/geometry/current-geometry.h5
Accurate geometry files are best obtained using the results of crystal indexing. Geometry refinement is extensively discussed on the CrystFEL web pages.
Peak mask (‘Do not analyse data here’)
Badly named variable called peakmask because it (initially) defined the regions to ignore for peak finding, for example jet streaks or shadows on the detector. Also used to define regions to be excluded from other analysis (single particle hit finding, radial averages, etc). Defines regions to ignore in analysis, but pass data through to the final image (as opposed to bad pixels, which are zeroed).
In the .ini file:
# Peakfinding region mask
peakmask=../../calib/mask/current-peakmask.h5
In practice this is pretty important: Shadowed pixels should not be included in radial averages, for example; while jet regions often result in false peaks and will also give bad values if you try to integrate reflectoins on top of the jet.
Peak mask regions are flagged in the pixel mask array using a different value so they can be treated differently in subsequent analysis (eg: do not integrate reflections in shadows or the jet region). Masks can be combined using the “Tools-->Combine masks” utility. Use ‘Shift’ to select the masks to be combined.
Detector gain
Some detectors enable different gain settings for different pixels.
In the .ini file:
# Gain calibration
gaincal=../../calib/gaincal/current-gaincal.h5
Default is no gain correction. Value in the file is the factor by which each pixel should be multiplied (eg: 7.2 for cspad regions in low-gain mode). Note: It may be necessary to change the data save format to provide sufficient data depth for the range of values in gain corrected data (beware integer overflow).
# Accepted formats = {INT16,INT32,float}
dataSaveFormat=INT16
Common mode
Talk to your local detector expert if you need to find out what to put in here
In the .ini file:
# Unbonded pixels only for cspad v1.6 modules (January 2014 onwards)
commonModeCorrection=none
#commonModeCorrection=asic_unbonded
Photon counting conversion
Talk to your local detector expert if you need to find out what to put in here
In the .ini file:
# Photon counting conversion
photonCount=1
photconv_ev=8600
photconv_adu=27
Noisy, hot or saturated pixels
Talk to your local detector expert if you need to find out what to put in here
In the .ini file:
#useAutoHotPixel=0
#hotpixADC=10000
#hotpixFreq=0.9
maskSaturatedPixels=1
pixelSaturationADC=16000
#useAutoNoisyPixel=1
#noisyPixMinDeviation=100
#noisyPixIncludeHits=1
#noisyPixRecalc=100
Per-pixel histograms for detector characterisation
Very useful for characterising the detector, for example single photon counting sensitivity, this produces a data cube representing the histogram of measured values through a whole run. Output is in the form of a data cube of shape corresponding to the detector array (raw layout, no geometry) by the number of histogram bins. Per-pixel histograms can easily take lots of memory so we perform a memory limit check to prevent chaos. Can be set to histogram only a region of the detector to get around this should you need lots of bins.
Look at:
/reg/g/cfel/cheetah/template/cheetah/process/histogram.ini
In the .ini file:
histogram=1
histogramMin=-50
histogramBinSize=1
histogramNbins=200
histogram_fs_min=0
histogram_fs_max=1552
histogram_ss_min=0
histogram_ss_max=1480
histogramMaxMemoryGb=4
hitfinder=0
Also, because the array is very large and takes a long time to save relative to calculation time:
saveInterval=10000
Pixel status
Pixel masks returned by Cheetah for each event contain useful information about the status determined of each pixel in a frame. This is very useful in your analysis: you can use bits in this mask to determine whether or not to include certain pixels in your analysis.
Admittedly some of the definitions admittedly appear a bit odd - probably because they were added for specific applications and remain for historic reasons.
Definitions in detectorObject.h:
/*
* Bits for pixel masks
* Oriented along conventions for CXI file format ( https://github.com/FilipeMaia/CXI/raw/master/cxi_file_format.pdf )
* CONVENTIONS:
* - All options are dominantly inherited during assembly and pixel integration (see assembleImage.cpp)
* - The default value for all options is "false"
*/
static const uint16_t PIXEL_IS_PERFECT = 0; // Change this value if necessary
static const uint16_t PIXEL_IS_INVALID = 1; // bit 0
static const uint16_t PIXEL_IS_SATURATED = 2; // bit 1
static const uint16_t PIXEL_IS_HOT = 4; // bit 2
static const uint16_t PIXEL_IS_DEAD = 8; // bit 3
static const uint16_t PIXEL_IS_SHADOWED = 16; // bit 4
static const uint16_t PIXEL_IS_IN_PEAKMASK = 32; // bit 5
static const uint16_t PIXEL_IS_TO_BE_IGNORED = 64; // bit 6
static const uint16_t PIXEL_IS_BAD = 128; // bit 7
static const uint16_t PIXEL_IS_OUT_OF_RESOLUTION_LIMITS = 256; // bit 8
static const uint16_t PIXEL_IS_MISSING = 512; // bit 9
static const uint16_t PIXEL_IS_NOISY = 1024; // bit 10
static const uint16_t PIXEL_IS_ARTIFACT_CORRECTED = 2048; // bit 11
static const uint16_t PIXEL_FAILED_ARTIFACT_CORRECTION = 4096; // bit 12
static const uint16_t PIXEL_IS_PEAK_FOR_HITFINDER = 8192; // bit 13
static const uint16_t PIXEL_IS_PHOTON_BACKGROUND_CORRECTED = 16384; // bit 14
static const uint16_t PIXEL_IS_IN_JET = 32768; // bit 15