Single frame files (2009-2014)

From 2009 - 2014 we output a single HDF5 file per image with unique file name.


This is incredibly convenient for small amounts of data; however single frame files rapidly destroy file system once the number of images becomes millions, or tens of millions, or more.  Single files effectively uses the file system as a metadata catalogue, and almost guarantee that your file system will be fragmented.  Copying many small files takes much more time than a smaller number of large files.  You can get away with it for small data sets, but it does not scale.  Trust us... we found out.  



This format is no longer the default.  You can roll back to this format by including the following line in the .ini file

    saveCXI=0

 

Are simple and almost self explanatory:


> h5ls -r LCLS_2011_Feb20_r0286_004938_5ad2.h5

/                        Group

/LCLS                    Group

/LCLS/detector0-EncoderValue Dataset {1}

/LCLS/detector0-Position Dataset {1}

/LCLS/ebeamCharge        Dataset {1}

/LCLS/ebeamL3Energy      Dataset {1}

/LCLS/ebeamLTUAngX       Dataset {1}

/LCLS/ebeamLTUAngY       Dataset {1}

/LCLS/ebeamLTUPosX       Dataset {1}

/LCLS/ebeamLTUPosY       Dataset {1}

/LCLS/ebeamPkCurrBC2     Dataset {1}

/LCLS/eventTime          Soft Link {/LCLS/eventTimeString}

/LCLS/eventTimeString    Dataset {SCALAR}

/LCLS/evr41              Dataset {1}

/LCLS/f_11_ENRC          Dataset {1}

/LCLS/f_12_ENRC          Dataset {1}

/LCLS/f_21_ENRC          Dataset {1}

/LCLS/f_22_ENRC          Dataset {1}

/LCLS/fiducial           Dataset {1}

/LCLS/machineTime        Dataset {1}

/LCLS/phaseCavityCharge1 Dataset {1}

/LCLS/phaseCavityCharge2 Dataset {1}

/LCLS/phaseCavityTime1   Dataset {1}

/LCLS/phaseCavityTime2   Dataset {1}

/LCLS/photon_energy_eV   Dataset {1}

/LCLS/photon_wavelength_A Dataset {1}

/data                    Group

/data/data               Soft Link {/data/rawdata0}

/data/pixelmask0         Dataset {1480, 1552}

/data/radialAverage0     Dataset {1175}

/data/radialAverageCounter0 Dataset {1175}

/data/rawdata            Soft Link {/data/rawdata0}

/data/rawdata0           Dataset {1480, 1552}

/processing              Group

/processing/cheetah      Group

/processing/cheetah/peakinfo Soft Link {/processing/hitfinder/peakinfo-raw}

/processing/cheetah/peakinfo-assembled Dataset {22, 4}

/processing/cheetah/peakinfo-raw Dataset {22, 4}

/processing/hitfinder    Group, same as /processing/cheetah


 

Single frame Cheetah files