Back to Cheetah at EuXFEL

Treat the AGIPD calibration data with as much care as your ‘normal’ data.  Bad calibration constants are perfectly capable of ruining otherwise good data, or at a minimum turning useable pixels into useless ones.

It is important to have a set of calibration files which matches the current pulse train and memory cell layout used at SFX/SPB, ideally obtained close to your experiment date as the detector ‘settles in’.

Calibration constants are needed for each gain stage of each memory cell. 

You can generate these files however you like: the above instructions are just one way of doing it.

Calibration files are referenced in the .ini file as ‘darkcal’ files:

    > grep darkcal  process/lys-agipd.ini

    darkcal=../../calib/agipd/Cheetah-AGIPD00-calib.h5

Specify only the AGIPD00 file (the others will be numbered automatically)

Calibration data is provided in a set of HDF5 calibration files with the following structure.

    > h5ls calib/agipd/Cheetah-AGIPD00-calib.h5

    AnalogOffset             Dataset {3, 32, 512, 128} H5T_STD_I16LE

    Badpixel                 Dataset {3, 32, 512, 128} H5T_STD_U8LE

    DigitalGainLevel         Dataset {3, 32, 512, 128} H5T_STD_U16LE

    RelativeGain             Dataset {3, 32, 512, 128} H5T_IEEE_F32LE

The dimensions of this array are {ngain, ncells, nss, nfs} where:

    ngain = number of gain stages (=3)

    ncells = number of memory cells in use (can vary)

    nss, nfs = number of pixels in slow- and fast-scan axes of the raw data, matching the order in XFEL DAQ layout.

Only the ncells dimension should change for different pulse patterns. nss and nfs are determined by the module size, ngain by the detector design.

There is one calibration file per module.  This is because each AGIPD module is saved into a separate RAW file by the EuXFEL DAQ.   Each calibration thus matches up to each RAW file.  There are 16 modules in the AGIPD-1M detector, thus there will be 16 calibration files named AGIPD00 to AGIPD15.  There is an example in the template (look in calib/agipd/*.h5)

Cheetah determines which memory cell to use for a given data frame from the CellId field of the image portion of the RAW data file.  You will therefore need a calibration file with at least as many entries as the number of memory cells in use for your data.  It doesn’t matter if the calibration file has more memory cells, but fewer memory cells than your data will result in an error.  You should also make sure that the calibration file was made using the same pulse pattern logic as your data - changes in vetoing strategy can change the physical memory cell numbering, for example.

Scripts for generating AGIPD calibration files for Cheetah are found in the calib/agipd/scripts directory of the Cheetah template at EuXFEL.

The procedure is as follows:

1. 1.Run the script preprocess-dk
   

> ./preprocess-dk <HG-dark-run> <MG-dark-run> <LG-dark-run>

eg

> ./preprocess-dk 1 2 3

Make sure to change the raw data path in the script.

    #!/bin/bash

    #

    # Script to run FS-DS calibrations

    # https://github.com/AGIPD/agipd-toolbox

    #

    # For new experiments, update the input_dir to new experiment

    #

    # Once set up, call as

    # preprocess-dk HG-run MG-run LG-run

    source /etc/profile.d/modules.sh

    source /gpfs/cfel/cxi/common/cfelsoft-rh7/setup.sh

    module load cfel-python3/latest

    # Run list needs to be HG MG LG in that order

    echo "Will run fs-ds scripts on AGIPD dark trio"

    echo "HG: " $1

    echo "MG: " $2

    echo "LG: " $3

    sleep 5

    /gpfs/cfel/cxi/scratch/user/barty/agipd_calib/agipd_tools/jenny/agipd-toolbox/calibration/job_scripts/sbatch_analyse.py \

    --input_dir /gpfs/exfel/exp/SPB/201801/p002120/ \

    --output_dir . \

    --type dark \

    --run_list $1 $2 $3  \

    --run_type all

2. Run the quick_calib.pro script

IDL> .run quick_calib.pro

This to turns the above data into Cheetah format and at the same time generates bad pixel masks for each memory cell.

Use the dialog box to select the dark calibration file generated in step 1.  Output will be placed automatically in a directory in the current working directory. Make sure to look over the results summaries to make sure they make sense!

This script is written in IDL, so you’ll have to load the IDL module and start IDL before running it.  I thought it would be used for only one experiment, but it’s lived on beyond that.  We may rewrite to remove this dependency at some point.

3. 3.Copy calibration files to the cheetah/calib/agipd directory
   

This is where we usually put calibration files.

Also create and update a new calib.ini file.  You’re then ready to go.

Caveats:

1. -Sometimes the dark runs are not good.  If you use ‘corrupted’ or ‘not dark’ runs in the pipeline, it’s going to fail.  Sometimes by crashing (check error files), sometimes by making all pixels bad, or by calculating wrong offsets (for example if the shutter was actually open).  Keep your eyes open and think.  That’s why it’s not fully automated yet.
   

2. -This process creates a simple calibration file using dark runs as the input.  It seems to do surprisingly well and is usually good enough.  One may be able to improve further by incorporating pulse capacitor gain calibration, but that procedure is much more complicated....
   

3. -Calibration constants appear to be fairly stable, but the bad pixel mask needs updating more often. This procedure does both at the same time.  It’s possible the main differences between calibrations is the per-memory-cell bad pixel mask.

AGIPD dynamically changes gain depending on the measured intensity for each frame.  There are two data channels: the usual ADC value, and an additional channel indicating which gain stage the pixel is in on that data frame.  We therefore need the following constants:

    GainLevel:  The signal level to use to determine which of the three gain stages is used, for each of the three gains

    AnalogOffset: The dark offset for each of the three gain stages

    RelativeGain: The gain (1/slope) of the response for each of the three gain stages. 

This is a total of 9 constants, per memory cell of each pixel.  Hence the number of fields in the calibration file.

Determining good calibration constants turns out to be critical. Put another way, wrong calibration values can turn useful pixels into bad values or even entirely useless pixels.  Incorrectly identifying the gain stage actually in use will result in nonsensical values of -68,000 or +1e6 or worse, and bad RelativeGain values can result in division by zero.  Make sure to do quality control on your calibration constants. 

The nominal gain values are approximately:

    High Gain              (1x)

    Medium Gain        (36x)

    Low gain               (180x)

Bad pixels can be identified from the calibration constants. For example:

1. -analog offset[gain, cell, pixel] is more than 4-5 standard deviations outside the module median value;
   

2. -gainLevel[gain, cell, pixel] is less than gainLevel[gain-1, cell, pixel]
   

Event synchronisation

All 16 AGIPD modules are saved into separate HDF5 files.  Cheetah synchronises events across all 16 files by comparing trainId and pulseId pairs to make sure that the same event is being read from all files.  The high indexing rate (~100% for Lysozyme) indicates this is being done successfully.  If the DAQ glitches for one module, the whole train may be missing from the HDF5 file resulting in in frames being out of synchronisation across the 16 AGIPD00-AGIPD15 files.  Furthermore, on occasion a module may ‘freeze’ and stop being saved until the DAQ is restarted, in which case data files for that module are simply not present.  While file synchronisation may be solved in the future, for the moment matching train and pulse IDs to assemble a complete event is an important and necessary function.

You can ask the XFEL detector group about calibration .  Or possibly the DESY FS-DS detector group if you are nice and include them in the experiment.  Since both groups use different formats you may have to write a converter to assemble data into the right form for Cheetah.  Currently XFEL and FS-DS use different layouts of HDF5 file, and even different array ordering, so post-processing their data files is needed. The same script can identify bad pixels on a per-cell and per-gain level and place this in the calibration file (==0 is a good pixel, !=0 is a bad pixel). The normal Cheetah bad pixel mask is applied in addition (but uses the opposite convention, ==0 is bad, ==1 is good, sorry about that).