Single CXI file (2014-2017)
From 2014 - 2017 we output a single large HDF5 file following the specification of the Coherent X-ray Imaging Data Bank (CXIDB). One huge file is friendly on the file system and copying smaller number of large files takes much less time than copying many small files.
However, everything is in the one file and rapidly becomes very large. This becomes inconvenient if one only wanted to do analysis of metadata, or perform any analysis without needing the frame data: for example, working with the time tool data in preparation for indexing, or working with radial averavges. Often data was saved in cryptic locations inn order to comply with the CXI (and Nexus) file specification which were inteded to handle detector data rather than large quantities of metadata.
This format is no longer the default. You can roll back to this format by including the following line in the .ini file
saveCXI=1
cxiLegacyFileFormat=2015
CXI file format definition
We follow the format of the Coherent X-ray Imaging Data Bank (CXIDB) which is loosely based on NeXus.
CXI files are HDF5 files with a pre-defined internal format. HDF5 can be read directly by any decent scientific data analysis software. You need to know what’s where since HDF5 is a container for multiple data sets.
Useful:
-
> h5ls -h
-
usage: h5ls [OPTIONS] file[/OBJECT] [file[/[OBJECT]...]
-
....
Follow the format of the Coherent X-ray Imaging Data Bank (CXIDB).
These files rapidly become complex (hence the simplified format in 2017)
> h5ls -r mfxn1116-r0062-c00.cxi
/ Group
/LCLS Group
/LCLS/detector_1 Group
/LCLS/detector_1/EncoderValue Dataset {92/Inf}
/LCLS/detector_1/SolidAngleConst Dataset {92/Inf}
/LCLS/detector_1/position Dataset {92/Inf}
/LCLS/ebeamCharge Dataset {92/Inf}
/LCLS/ebeamL3Energy Dataset {92/Inf}
/LCLS/ebeamLTUAngX Dataset {92/Inf}
/LCLS/ebeamLTUAngY Dataset {92/Inf}
/LCLS/ebeamLTUPosX Dataset {92/Inf}
/LCLS/ebeamLTUPosY Dataset {92/Inf}
/LCLS/ebeamPkCurrBC2 Dataset {92/Inf}
/LCLS/eventTime Soft Link {eventTimeString}
/LCLS/eventTimeString Dataset {92/Inf}
/LCLS/evr41 Dataset {92/Inf}
/LCLS/experiment_identifier Soft Link {/entry_1/experiment_identifier}
/LCLS/f_11_ENRC Dataset {92/Inf}
/LCLS/f_12_ENRC Dataset {92/Inf}
/LCLS/f_21_ENRC Dataset {92/Inf}
/LCLS/f_22_ENRC Dataset {92/Inf}
/LCLS/fiducial Dataset {92/Inf}
/LCLS/machineTime Dataset {92/Inf}
/LCLS/machineTimeNanoSeconds Dataset {92/Inf}
/LCLS/phaseCavityCharge1 Dataset {92/Inf}
/LCLS/phaseCavityCharge2 Dataset {92/Inf}
/LCLS/phaseCavityTime1 Dataset {92/Inf}
/LCLS/phaseCavityTime2 Dataset {92/Inf}
/LCLS/photon_energy_eV Dataset {92/Inf}
/LCLS/photon_wavelength_A Dataset {92/Inf}
/cheetah Group
/cheetah/configuration Group
/cheetah/configuration/input Dataset {SCALAR}
/cheetah/event_data Group
/cheetah/event_data/detector_1 Group
/cheetah/event_data/detector_1/sum Dataset {92/Inf}
/cheetah/event_data/energySpectrumExist Dataset {92/Inf}
/cheetah/event_data/eventName Dataset {92/Inf}
/cheetah/event_data/frameNumber Dataset {92/Inf}
/cheetah/event_data/frameNumberIncludingSkipped Dataset {92/Inf}
/cheetah/event_data/gmd1 Dataset {92/Inf}
/cheetah/event_data/gmd2 Dataset {92/Inf}
/cheetah/event_data/hit Dataset {92/Inf}
/cheetah/event_data/imageClass Dataset {92/Inf}
/cheetah/event_data/nPeaks Dataset {92/Inf}
/cheetah/event_data/nProtons Dataset {92/Inf}
/cheetah/event_data/peakDensity Dataset {92/Inf}
/cheetah/event_data/peakNpix Dataset {92/Inf}
/cheetah/event_data/peakResolution Dataset {92/Inf}
/cheetah/event_data/peakResolutionA Dataset {92/Inf}
/cheetah/event_data/peakTotal Dataset {92/Inf}
/cheetah/event_data/pumpLaserCode Dataset {92/Inf}
/cheetah/event_data/pumpLaserDelay Dataset {92/Inf}
/cheetah/event_data/threadID Dataset {92/Inf}
/cheetah/global_data Group
/cheetah/global_data/detector_1 Group
/cheetah/global_data/detector_1/class_1 Group
/cheetah/global_data/detector_1/class_1/mean_non_assembled_detector_and_photon_corrected Dataset {1480, 1552}
/cheetah/global_data/detector_1/class_1/mean_non_assembled_detector_corrected Dataset {1480, 1552}
/cheetah/global_data/detector_1/class_1/mean_radial_average_detector_and_photon_corrected Dataset {1167}
/cheetah/global_data/detector_1/class_1/mean_radial_average_detector_corrected Dataset {1167}
/cheetah/global_data/detector_1/class_1/sigma_non_assembled_detector_and_photon_corrected Dataset {1480, 1552}
/cheetah/global_data/detector_1/class_1/sigma_non_assembled_detector_corrected Dataset {1480, 1552}
/cheetah/global_data/detector_1/class_1/sigma_radial_average_detector_and_photon_corrected Dataset {1167}
/cheetah/global_data/detector_1/class_1/sigma_radial_average_detector_corrected Dataset {1167}
/cheetah/global_data/detector_1/class_2 Group
/cheetah/global_data/detector_1/class_2/mean_non_assembled_detector_and_photon_corrected Dataset {1480, 1552}
/cheetah/global_data/detector_1/class_2/mean_non_assembled_detector_corrected Dataset {1480, 1552}
/cheetah/global_data/detector_1/class_2/mean_radial_average_detector_and_photon_corrected Dataset {1167}
/cheetah/global_data/detector_1/class_2/mean_radial_average_detector_corrected Dataset {1167}
/cheetah/global_data/detector_1/class_2/sigma_non_assembled_detector_and_photon_corrected Dataset {1480, 1552}
/cheetah/global_data/detector_1/class_2/sigma_non_assembled_detector_corrected Dataset {1480, 1552}
/cheetah/global_data/detector_1/class_2/sigma_radial_average_detector_and_photon_corrected Dataset {1167}
/cheetah/global_data/detector_1/class_2/sigma_radial_average_detector_corrected Dataset {1167}
/cheetah/global_data/detector_1/hotPixBufferCounter Dataset {92/Inf}
/cheetah/global_data/detector_1/lastBgUpdate Dataset {92/Inf}
/cheetah/global_data/detector_1/lastHotPixUpdate Dataset {92/Inf}
/cheetah/global_data/detector_1/lastNoisyPixUpdate Dataset {92/Inf}
/cheetah/global_data/detector_1/nHot Dataset {92/Inf}
/cheetah/global_data/detector_1/nNoisy Dataset {92/Inf}
/cheetah/global_data/detector_1/noisyPixBufferCounter Dataset {92/Inf}
/cheetah/global_data/hit Dataset {92/Inf}
/cheetah/global_data/nPeaks Dataset {92/Inf}
/cheetah_version_commit Dataset {SCALAR}
/cxi_version Dataset {1}
/entry_1 Group
/entry_1/data_1 Soft Link {/entry_1/instrument_1/detector_1}
/entry_1/experiment_identifier Dataset {92/Inf}
/entry_1/image_1 Group
/entry_1/image_1/data Soft Link {/entry_1/image_1/detector_corrected/data}
/entry_1/image_1/data_space Soft Link {/entry_1/image_1/detector_corrected/data_space}
/entry_1/image_1/data_type Soft Link {/entry_1/image_1/detector_corrected/data_type}
/entry_1/image_1/detector_1 Soft Link {/entry_1/instrument_1/detector_1}
/entry_1/image_1/detector_corrected Group
/entry_1/image_1/detector_corrected/data Dataset {92/Inf, 1167}
/entry_1/image_1/detector_corrected/data_space Dataset {92/Inf}
/entry_1/image_1/detector_corrected/data_type Dataset {92/Inf}
/entry_1/image_1/detector_corrected/experiment_identifier Soft Link {/entry_1/experiment_identifier}
/entry_1/image_1/detector_corrected/mask Dataset {92/Inf, 1167}
/entry_1/image_1/detector_corrected/mask_shared Dataset {1167}
/entry_1/image_1/experiment_identifier Soft Link {/entry_1/experiment_identifier}
/entry_1/image_1/mask Soft Link {/entry_1/image_1/detector_corrected/mask}
/entry_1/image_1/mask_shared Soft Link {/entry_1/image_1/detector_corrected/mask_shared}
/entry_1/image_1/source_1 Soft Link {/entry_1/instrument_1/source_1}
/entry_1/instrument_1 Group
/entry_1/instrument_1/detector_1 Group
/entry_1/instrument_1/detector_1/data Soft Link {/entry_1/instrument_1/detector_1/detector_corrected/data}
/entry_1/instrument_1/detector_1/description Dataset {SCALAR}
/entry_1/instrument_1/detector_1/detector_corrected Group
/entry_1/instrument_1/detector_1/detector_corrected/data Dataset {92/Inf, 1480, 1552}
/entry_1/instrument_1/detector_1/detector_corrected/experiment_identifier Soft Link {/entry_1/experiment_identifier}
/entry_1/instrument_1/detector_1/detector_corrected/mask Dataset {92/Inf, 1480, 1552}
/entry_1/instrument_1/detector_1/detector_corrected/mask_shared Dataset {1480, 1552}
/entry_1/instrument_1/detector_1/detector_corrected/mask_shared_max Dataset {1480, 1552}
/entry_1/instrument_1/detector_1/detector_corrected/mask_shared_min Dataset {1480, 1552}
/entry_1/instrument_1/detector_1/distance Dataset {92/Inf}
/entry_1/instrument_1/detector_1/experiment_identifier Soft Link {/entry_1/experiment_identifier}
/entry_1/instrument_1/detector_1/mask Soft Link {/entry_1/instrument_1/detector_1/detector_corrected/mask}
/entry_1/instrument_1/detector_1/mask_shared Soft Link {/entry_1/instrument_1/detector_1/detector_corrected/mask_shared}
/entry_1/instrument_1/detector_1/mask_shared_max Soft Link {/entry_1/instrument_1/detector_1/detector_corrected/mask_shared_max}
/entry_1/instrument_1/detector_1/mask_shared_min Soft Link {/entry_1/instrument_1/detector_1/detector_corrected/mask_shared_min}
/entry_1/instrument_1/detector_1/x_pixel_size Dataset {92/Inf}
/entry_1/instrument_1/detector_1/y_pixel_size Dataset {92/Inf}
/entry_1/instrument_1/source_1 Group
/entry_1/instrument_1/source_1/energy Dataset {92/Inf}
/entry_1/instrument_1/source_1/experiment_identifier Soft Link {/entry_1/experiment_identifier}
/entry_1/result_1 Group
/entry_1/result_1/data Soft Link {peakTotalIntensity}
/entry_1/result_1/nPeaks Dataset {92/Inf}
/entry_1/result_1/peakMaximumValue Dataset {92/Inf, 1024/Inf}
/entry_1/result_1/peakNPixels Dataset {92/Inf, 1024/Inf}
/entry_1/result_1/peakSNR Dataset {92/Inf, 1024/Inf}
/entry_1/result_1/peakTotalIntensity Dataset {92/Inf, 1024/Inf}
/entry_1/result_1/peakXPosAssembled Dataset {92/Inf, 1024/Inf}
/entry_1/result_1/peakXPosRaw Dataset {92/Inf, 1024/Inf}
/entry_1/result_1/peakYPosAssembled Dataset {92/Inf, 1024/Inf}
/entry_1/result_1/peakYPosRaw Dataset {92/Inf, 1024/Inf}
/entry_1/result_1/powderClass Dataset {92/Inf}
/psana_version_commit Dataset {SCALAR}
Single large CXI file