Current file format (2017+)
Image data saved into a CXI format compliant file (*.cxi)
Processing results are saved into a separate HDF5 file (*.h5)
Events are synchronised between both files (same stack position in both files corresponds to the same event)
Results files are much smaller than CXI files because they do not contain the entire image for each event:
-rw-rw----+ 1 barty xu 13G May 15 02:39 mfxn1116-r0040-c00.cxi
-rw-rw----+ 1 barty xu 269M May 15 02:39 mfxn1116-r0040-c00.h5
This has several benefits:
-
• Benefits of a small number of large files are retained
-
• Results file can be more flexibly defined to suit FEL data without having to force data into the Nexus standard
-
• Results and metadata can be worked on without needing to copy the (large) .cxi file
-
• Image file complies more closely to standard CXI and Nexus files
HDF5 is a well supported industry standard container data format that can be directly read by any decent scientific data analysis software.
Useful:
-
> h5ls -h
-
usage: h5ls [OPTIONS] file[/OBJECT] [file[/[OBJECT]...]
-
....
REsults file (.h5)
Broadly speaking:
/instrument/... = data read directly from the instrument (encoder values, gas detectors, photon energy, etc)
/event_data = results which appear for each event (peak lists, radial average stack, etc)
/run_data = results accumulated across the entire run (detector sums, virtual powder, pixel histograms)
/cheetah/... = things pertaining to cheetah internals (configuration, threadID, event number, etc)
Data locations are designed to be discoverable using the h5ls command, for example:
> h5ls -r cxij4716-r0059-c00.h5/instrument
/detector0 Group
/detector0/EncoderValue Dataset {524/Inf}
/detector0/position Dataset {524/Inf}
/ebeamCharge Dataset {524/Inf}
/ebeamL3Energy Dataset {524/Inf}
/ebeamLTUAngX Dataset {524/Inf}
/ebeamLTUAngY Dataset {524/Inf}
/ebeamLTUPosX Dataset {524/Inf}
/ebeamLTUPosY Dataset {524/Inf}
/ebeamPkCurrBC2 Dataset {524/Inf}
/eventTimeString Dataset {524/Inf}
> h5ls -r cxij4716-r0059-c00.h5/run_data
/detector0 Group
/detector0/class1 Group
/detector0/class1/mean_non_assembled_detector_and_photon_corrected Dataset {1480, 1552}
/detector0/class1/mean_non_assembled_detector_corrected Dataset {1480, 1552}
/detector0/class1/mean_radial_average_detector_and_photon_corrected Dataset {1187}
/detector0/class1/mean_radial_average_detector_corrected Dataset {1187}
/detector0/class1/sigma_non_assembled_detector_and_photon_corrected Dataset {1480, 1552}
/detector0/class1/sigma_non_assembled_detector_corrected Dataset {1480, 1552}
/detector0/class1/sigma_radial_average_detector_and_photon_corrected Dataset {1187}
/detector0/class1/sigma_radial_average_detector_corrected Dataset {1187}
Or using an interactive HDF5 file browser as shown below:
Image data is saved in the format of the Coherent X-ray Imaging Data Bank (CXIDB) which is loosely based on NeXus.
A small amount of results are saved so as to not immediately break compatibility with CrystFEL
> h5ls -r mfxn1116-r0040-c00.cxi
/ Group
/LCLS Soft Link {instrument}
/entry_1 Group
/entry_1/data0 Soft Link {/entry_1/instrument_1/detector_1}
/entry_1/experiment_identifier Dataset {1959/Inf}
/entry_1/instrument_1 Group
/entry_1/instrument_1/detector_1 Group
/entry_1/instrument_1/detector_1/data Soft Link {/entry_1/instrument_1/detector_1/detector_corrected/data}
/entry_1/instrument_1/detector_1/description Dataset {SCALAR}
/entry_1/instrument_1/detector_1/detector_corrected Group
/entry_1/instrument_1/detector_1/detector_corrected/data Dataset {1959/Inf, 1480, 1552}
/entry_1/instrument_1/detector_1/detector_corrected/experiment_identifier Soft Link {/entry_1/experiment_identifier}
/entry_1/instrument_1/detector_1/detector_corrected/mask Dataset {1959/Inf, 1480, 1552}
/entry_1/instrument_1/detector_1/detector_corrected/mask_shared Dataset {1480, 1552}
/entry_1/instrument_1/detector_1/detector_corrected/mask_shared_max Dataset {1480, 1552}
/entry_1/instrument_1/detector_1/detector_corrected/mask_shared_min Dataset {1480, 1552}
/entry_1/instrument_1/detector_1/distance Dataset {1959/Inf}
/entry_1/instrument_1/detector_1/experiment_identifier Soft Link {/entry_1/experiment_identifier}
/entry_1/instrument_1/detector_1/mask Soft Link {/entry_1/instrument_1/detector_1/detector_corrected/mask}
/entry_1/instrument_1/detector_1/mask_shared Soft Link {/entry_1/instrument_1/detector_1/detector_corrected/mask_shared}
/entry_1/instrument_1/detector_1/mask_shared_max Soft Link {/entry_1/instrument_1/detector_1/detector_corrected/mask_shared_max}
/entry_1/instrument_1/detector_1/mask_shared_min Soft Link {/entry_1/instrument_1/detector_1/detector_corrected/mask_shared_min}
/entry_1/instrument_1/detector_1/x_pixel_size Dataset {1959/Inf}
/entry_1/instrument_1/detector_1/y_pixel_size Dataset {1959/Inf}
/entry_1/instrument_1/experiment_identifier Soft Link {/entry_1/experiment_identifier}
/entry_1/instrument_1/source_1 Group
/entry_1/instrument_1/source_1/energy Dataset {1959/Inf}
/entry_1/instrument_1/source_1/experiment_identifier Soft Link {/entry_1/experiment_identifier}
/entry_1/result_1 Group
/entry_1/result_1/data Soft Link {peakTotalIntensity}
/entry_1/result_1/nPeaks Dataset {1959/Inf}
/entry_1/result_1/peakMaximumValue Dataset {1959/Inf, 1024/Inf}
/entry_1/result_1/peakNPixels Dataset {1959/Inf, 1024/Inf}
/entry_1/result_1/peakSNR Dataset {1959/Inf, 1024/Inf}
/entry_1/result_1/peakTotalIntensity Dataset {1959/Inf, 1024/Inf}
/entry_1/result_1/peakXPosAssembled Dataset {1959/Inf, 1024/Inf}
/entry_1/result_1/peakXPosRaw Dataset {1959/Inf, 1024/Inf}
/entry_1/result_1/peakYPosAssembled Dataset {1959/Inf, 1024/Inf}
/entry_1/result_1/peakYPosRaw Dataset {1959/Inf, 1024/Inf}
/entry_1/result_1/powderClass Dataset {1959/Inf}
/instrument Group
/instrument/detector_1 Group
/instrument/detector_1/EncoderValue Dataset {1959/Inf}
/instrument/detector_1/SolidAngleConst Dataset {1959/Inf}
/instrument/detector_1/position Dataset {1959/Inf}
/instrument/eventTimeString Dataset {1959/Inf}
/instrument/fiducial Dataset {1959/Inf}
/instrument/machineTime Dataset {1959/Inf}
/instrument/photon_energy_eV Dataset {1959/Inf}
/instrument/photon_wavelength_A Dataset {1959/Inf}
IMage data file (.CXI)