CBOR messages

To communicate between FPGA-equipped receiver system and writers, Jungfraujoch is using binary CBOR encoding with tinycbor library (Intel). The protocol is based on and compatible with DECTRIS Stream2. There are minor differences at the moment:

• Deserialization works only for single channel images

• LZ4 compression is not allowed; Bitshuffle Zstandard is possible

• Few fields are currently absent

• Extra fields are present beyond DECTRIS standard

• There are calibration and metadata messages defined beyond DECTRIS specification

Start message

Field name	Type	Description	Present in DECTRIS format
type	String	value “start”	X
magic_number	uint64	Number used to describe version of the Jungfraujoch data interface - to allow to detect inconsistency between sender and receiver
detector_distance	float	Detector distance [m]
detector_translation	float	Detector translation vector [m]	X
beam_center_x	float	Beam center in X direction [pixels]	X
beam_center_y	float	Beam center in Y direction [pixels]	X
countrate_correction_enabled	bool	Countrate correction enabled	X
flatfield_enabled	bool	Flatfield enabled	X
number_of_images	uint64	Number of images in the series	X
image_size_x	uint64	Image width [pixels]	X
image_size_y	uint64	Image height [pixels]	X
incident_energy	float	X-ray energy [eV]	X
incident_wavelength	float	X-ray wavelength [Angstrom]	X
frame_time	float	Frame time, if multiple frames per trigger [s]	X
count_time	float	Exposure time [s]	X
saturation_value	int64	Maximum valid sample value	X
error_value	int64 (optional)	Value used in images to describe pixels that are in error state or missing
pixel_size_x	float	Pixel width [m]	X
pixel_size_y	float	Pixel height [m]	X
sensor_thickness	float	Sensor thickness [m]	X
sensor_material	string	Sensor material	X
arm_data	data	Approximate date of arming	X
pixel_mask_enabled	bool	Pixel mask applied on images	X
detector_description	string	Name of the detector	X
detector_serial_number	string	Detector serial number	X
series_unique_id	string	Unique text ID of the series (run_name parameter)	X
series_id	uint64	Unique numeric ID of the series (run_number parameter)	X
goniometer	Map	See DECTRIS documentation - only one axis is supported	X
jungfrau_conversion_enabled	bool (optional)	Applying JUNGFRAU pixel conversion (to photons or keV)
jungfrau_conversion_factor	float (optional)	Factor used for JUNGFRAU conversion [eV]
geometry_transformation_enabled	bool (optional)	Transformation from detector module geometry (512x1024) to full detector geometry
pixel_mask	Map(string -> Image)	Pixel mask - multiple in case of storage cells	X
channels	Array(string)	List of image channels	X
max_spot_count	uint64	Maximum number of spots identified in spot finding
storage_cell_number	uint64 (optional)	Number of storage cells used by JUNGFRAU
storage_cell_delay	Rational	Delay of storage cells in JUNGFRAU
image_dtype	string	Pixel bit type (e.g. uint16)	X
unit_cell	object (optional)	Unit cell of the system: a, b, c [angstrom] and alpha, beta, gamma [degree]
az_int_bin_number	uint64	Number of azimuthal integration bins
az_int_bin_to_q	Array(float)	Q value for each azimuthal integration bin [angstrom^-1]
summation	uint64	Factor of frame summation
user_data	string	JSON serialized to string that can contain the following fields:	X
- file_prefix	string	File prefix
- images_per_file	string	Number of images written per file
- source_name	string	Facility name
- source_type	string (optional)	Type of X-ray source (use NXsource/type values, for example “Synchrotron X-ray Source” or “Free-Electron Laser”)
- instrument_name	string	Instrument name
- sample_name	string	Name of the sample
- user	any valid JSON (optional)	Value of header_appendix provided at collection start to Jungfraujoch
- attenuator_transmission	float (optional)	Attenuator transmission []
- total_flux	float (optional)	Total flux [ph/s]
- rotation_axis	Array(float)	Rotation axis vector (array of size 3)
- space_group_number	uint64	Space group number
- summation_mode	string	Summation mode (internal|fpga|cpu)
- overwrite	bool (optional)	Overwrite existing HDF5 files
- hdf5_format_version	int (optional)	1 = use soft links to data file, 2 = use virtual data set
- roi_names	Array(string)	Names of ROI
- gain_file_names	Array(string) (optional)	Names of JUNGFRAU gain files used for the current detector
- write_master_file	bool (optional)	With multiple sockets, it selects which socket will provide master file
- experiment_group	string	ID of instrument user, e.g., p-group (SLS/SwissFEL) or proposal number
- jfjoch_release	string (optional)	Jungfraujoch release number
- bit_depth_readout	uint64 (optional)	Bit depth of the detector readout
- writer_notification_zmq_addr	string (optional)	ZeroMQ address to inform jfjoch_broker about writers that finished operation
- socket_number	uint64 (optional)	Number of ZeroMQ socket (on jfjoch_broker side) used for transmission

See DECTRIS documentation for definition of Image as MultiDimArray with optional compression.

Image message

Field name	Type	Description	Present in DECTRIS format	Optional
type	String	value “image”	X
magic_number	uint64	Number used to describe version of the Jungfraujoch data interface - to allow to detect inconsistency between sender and receiver
series_unique_id	string	Unique text ID of the series (run_name parameter)	X
series_id	uint64	Unique numeric ID of the series (run_number parameter)	X
image_id	uint64	Number of image within the series; for MX lossy compression this is sequential excluding removed frames	X
original_image_id	uint64	Number of image within the series; for MX lossy compression this includes removed frames in the count
real_time	Rational	Exposure time	X
start_time	Rational	Exposure start time (highly approximate)	X
end_time	Rational	Exposure end time (highly approximate)	X
spots	Array(object)	Spots:
- x	float	position in x (pixels)
- y	float	position in y (pixels)
- I	float	intensity (photons)
- indexed	bool	indexed solution
spot_count	uint64	Spot count
spot_count_in_rings	uint64	Number of spots within identified rings (experimental)
az_int_profile	Array(float)	Azimuthal integration results, use az_int_bin_to_q from start message for legend
indexing_result	bool	Indexing successful
indexing_lattice	Array(9 * float)	Indexing result real lattice; present only if indexed		X
indexing_unit_cell	object	Indexing result unit cell: a, b, c [angstrom] and alpha, beta, gamma [degree]; present only if indexed		X
		Unit cell is redundant to lattice - yet to simplify downstream programs to analyze results, both are provided
xfel_pulse_id	uint64	Bunch ID (for pulsed source, e.g., SwissFEL)		X
xfel_event_code	uint64	Event code (for pulsed source, e.g., SwissFEL)		X
jf_info	uint64	Detector info field
receiver_aq_dev_delay	uint64	Receiver internal delay
receiver_free_send_buf	uint64	Receiver internal number of available send buffers
storage_cell	uint64	Storage cell number
saturated_pixel_count	uint64	Saturated pixel count
error_pixel_count	uint64	Error pixel count
strong_pixel_count	uint64	Strong pixel count (first stage of spot finding)
min_viable_pixel_value	int64	Minimal pixel value, excl. error and saturation
max_viable_pixel_value	int64	Maximal pixel value, excl. error and saturation
data_collection_efficiency	float	Image collection efficiency []
packets_expected	uint64	Number of packets expected per image (in units of 2 kB)
packets_received	uint64	Number of packets received per image (in units of 2 kB)
bkg_estimate	float	Mean value for pixels in resolution range from 3.0 to 5.0 A [\photons]
adu_histogram	Array(uint64)	ADU histogram
roi_integrals	object	Results of ROI calculation		X
- sum	int64	Sum of pixels in ROI area [photons]
- sum_square	int64	Sum of squares of pixels in ROI area [photons]
- pixels	int64	Valid pixels in ROI area [photons]
- max_count	int64	Highest count in ROI area [photons]
user_data	string	Optional user defined text information - this is image_appendix serialized to JSON format	X
data	Map(string -> Image)	Image	X

Metadata message

Field name	Type	Description	Present in DECTRIS format	Optional
type	String	value “metadata”	X
magic_number	uint64	Number used to describe version of the Jungfraujoch data interface - to allow to detect inconsistency between sender and receiver
series_unique_id	string	Unique text ID of the series (run_name parameter)	X
series_id	uint64	Unique numeric ID of the series (run_number parameter)	X
images	Array(object)	Array of images (order and size of the array are not guaranteed)	X
- image_id	uint64	Number of image within the series; for MX lossy compression this is sequential excluding removed frames	X
- original_image_id	uint64	Number of image within the series; for MX lossy compression this includes removed frames in the count
- real_time	Rational	Exposure time	X
- start_time	Rational	Exposure start time (highly approximate)	X
- end_time	Rational	Exposure end time (highly approximate)	X
- spot_count	uint64	Spot count
- spot_count_in_rings	uint64	Number of spots within identified rings (experimental)
- az_int_profile	Array(float)	Azimuthal integration results, use az_int_bin_to_q from start message for legend
- indexing_result	bool	Indexing successful
- indexing_lattice	Array(9 * float)	Indexing result real lattice; present only if indexed		X
- indexing_unit_cell	object	Indexing result unit cell: a, b, c [angstrom] and alpha, beta, gamma [degree]; present only if indexed		X
		Unit cell is redundant to lattice - yet to simplify downstream programs to analyze results, both are provided
- xfel_pulse_id	uint64	Bunch ID (for pulsed source, e.g., SwissFEL)		X
- xfel_event_code	uint64	Event code (for pulsed source, e.g., SwissFEL)		X
- jf_info	uint64	Detector info field
- receiver_aq_dev_delay	uint64	Receiver internal delay
- receiver_free_send_buf	uint64	Receiver internal number of available send buffers
- storage_cell	uint64	Storage cell number
- saturated_pixel_count	uint64	Saturated pixel count
- error_pixel_count	uint64	Error pixel count
- strong_pixel_count	uint64	Strong pixel count (first stage of spot finding)
- data_collection_efficiency	float	Image collection efficiency []
- bkg_estimate	float	Mean value for pixels in resolution range from 3.0 to 5.0 A [\photons]
- adu_histogram	Array(uint64)	ADU histogram
- roi_integrals	object	Results of ROI calculation		X
- - sum	int64	Sum of pixels in ROI area [photons]
- - sum_square	int64	Sum of squares of pixels in ROI area [photons]
- - pixels	int64	Valid pixels in ROI area [photons]
- - max_count	int64	Highest count in ROI area [photons]

End message

Field name	Type	Description	Present in DECTRIS format
type	String	value “end”	X
magic_number	uint64	Number used to describe version of the Jungfraujoch data interface - to allow to detect inconsistency between sender and receiver
series_unique_id	string	Unique text ID of the series (run_name parameter)	X
series_id	uint64	Unique numeric ID of the series (run_number parameter)	X
end_date	date	Approximate end date
max_image_number	uint64	Number of image with the highest number (this is counted from 1 - to distinguish zero images and one image)
images_collected	uint64	Number of image collected
images_sent_to_writer	uint64	Number of image sent to writer; if writer queues were full, it is possible this is less than images collected
data_collection_efficiency	float	Network packets collected / Network packets expected []
az_int_result	Map(text->Array(float))	Azimuthal integration results, use az_int_bin_to_q from start message for legend
adu_histogram	Map(text->Array(uint64))	ADU values histogram
adu_histogram_width	uint64	Width of bins in the above histogram [ADU]
max_receiver_delay	uint64	Internal performance of Jungfraujoch
bkg_estimate	float	Mean background estimate for the whole run
indexing_rate	float	Mean indexing rate for the whole run

Calibration message

Field name	Type	Description	Present in DECTRIS format
type	String	value “end”
magic_number	uint64	Number used to describe version of the Jungfraujoch data interface - to allow to detect inconsistency between sender and receiver
data	Map(string -> Image)	Calibration map (only single pedestal array per message)

User data

In many cases there is an interest from facilities to forward more metadata, than available explicitly in the Jungfraujoch. For this reason two fields can be provided: header_appendix (sent with start message) and image_appendix (send with image message). To increase flexibility, both appendices can contain any valid JSON message. These appendices are serialized into string and stored in CBOR messages as user_data.

Notably for start message, user_data contain more information (non-DECTRIS compliant metadata). Therefore user_data is serialized by Jungfraujoch as CBOR object. There is member user which contains header_appendix defined in OpenAPI of Jungfraujoch.