Changelog¶

1.0.0¶

1.0.0-rc.160¶

This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use.

rugnux: Rotation geometry post-refinement is now on by default (--rotation-no-postrefine to disable; also a viewer checkbox). A first pass integrates at the header geometry, then the shared detector distance + beam centre and the crystal cell/goniometer-axis are post-refined over all frames (cross-validated, committed only for a small < 1 % move, with the gauge-weak beam centre restrained toward the header); a second pass re-indexes de novo and re-integrates at the refined geometry. The refined pass is the canonical <prefix>_* output; the header-geometry pass is kept as <prefix>_01_*.
rugnux: Optional per-batch relative-B correction for rotation (--relative-b[=deg], default 10°-of-rotation batches when bare, off otherwise) - a cross-validated, curvature-smoothed resolution×dose correction beyond the single global decay slope.
rugnux: Always-on radiation-damage report for rotation - the per-image scale correlation-to-merge and mosaicity versus dose, plus the relative B-factor change over the run (first→last) as a scalar and a per-batch relative-B curve, printed to the log and written to the merged mmCIF. Report-only; it never alters the merge.
rugnux: De-novo space-group search ranks candidate lattice centerings by net absences (systematically-absent minus violating), not the gross absent count, fixing an over-centering of a genuinely C-centred lattice to F.
rugnux: Record the producing software (name and version) and the refined detector distance and beam centre in the merged mmCIF (and the software in the MTZ history).
Bragg integration: Carry the box-sum observed centroid through the profile-fit path, so the observed spot centroid is emitted in every integrator mode.
rugnux: Report ISa as the counting-subtracted strong-reflection asymptote, not 1/b of the whole-range fit; it also sets the merged-sigma floor. CC1/2, R-meas and per-obs sigmas unchanged.
Frontend: Azimuthal-integration Q fields (Q spacing / Low Q / High Q) accept 5 decimals (was 3), matching the 1e-5 q_spacing minimum; number-field precision is now configurable.
rugnux: Add a dataset-wide Wilson B-factor estimate to the merged output (mmCIF, stats table, log); the per-image viewer Wilson B emits NaN for implausible fits.
rugnux: De-novo space-group search vetoes a merohedral-twin over-promotion whose systematic error-model b balloons past a calibrated bound (keeps R3 as R3, not R32).
rugnux: De-novo space-group search decides lattice centering from the strength (mean I/sigma) of the systematically-absent class, not a per-reflection violation count.
rugnux: Recover lattice centering on weak / low-energy data via a floor-independent test (rate of significant absent vs present reflections), fixing a missed I-centring at 5/13 keV.
rugnux: Report anomalous signal-to-noise SigAno = <|I(+)-I(-)|>/ per shell and overall (mmCIF PDBx items + stats-table column); anomalous merges only.
rugnux: De-novo space-group search recovers a genuine high-symmetry group on weak data with a broken sigma model by confirming on the systematic-b test alone (restores an F432 case).
rugnux: Print the adopted space group and unit cell as a one-line summary at the end of the run (de-novo or user-fixed -S).
rugnux: Score the radiation-damage decay cross-validation on a sigma-independent (R-meas-like) metric, so a spurious slope can’t pass by reshaping sigmas.
rugnux: Fix de-novo rotation indexing committing a spurious axis-multiple supercell (collapsing to P1) via a cross-scheme smaller-cell tie-break on near-integer volume ratios.
rugnux: Widen refined-cell angle bounds to [30, 150] deg (rotation candidate and per-frame stills refinement); check refined angles against the reference cell.
rugnux: -S/--space-group now accepts a Hermann-Mauguin symbol (e.g. P43212) as well as a space-group number.
rugnux: Warn when the chosen cell/space group carries an indexing (merohedral) ambiguity needing a reference to resolve.
Indexing: Requesting the FFTW (CPU) indexer on a GPU node now fails with a clear, actionable message (rotation always uses the GPU FFT indexer there).
rugnux: Stills --refine-geometry[=N|off] - first-pass bundle-adjust of beam/distance/cell then re-index (default ON with a reference cell); accepts reference F/FP columns.
rugnux: Per-image geometry refinement -r flex tries all three algorithms per image and keeps the best (old name multi kept as an alias).
rugnux: Experimental stills partiality --still-partiality (Gaussian excitation-error) and --partiality-uncertainty <num> down-weighting the least-complete partials.
rugnux: Default the stills Bragg-integration box to r=6 (integration radii 6, 8, 12).
rugnux: Self-referenced stills scale in a single pass (fixing a weak-data collapse); a reference MTZ (-z) fixes SG/cell/ambiguity but never anchors the scale (stills and rotation).
rugnux: Cap normalised intensity (E^2) on second-lattice overlaps in the de-novo space-group search, so strong overlaps don’t skew the symmetry decision.
rugnux: Fix --scale on a self-contained _process.h5 (stored reflections and error model reload correctly).
rugnux: Add --spot-low-resolution <num> (default 50 A) and --min-pix-per-spot <num> (default 2) to tune spot finding on weak serial data.
jfjoch_viewer: Expose stills processing settings in the settings dock, rename geometry-refinement multi to flex, and refit the initial image on resize.
jfjoch_writer: Remove the CBF and TIFF image writers - only NXmx HDF5 is written (all three layouts remain).
Reader: Treat a negative total_flux in a stored dataset as unknown/absent rather than a valid flux.
Packaging: Build the self-contained Linux viewer against a static libdbus with glib disabled; add parallel image-build and in-container viewer-verification scripts.
rugnux: Write anomalous data as a standard CCP4 anomalous MTZ (one row per reflection: IMEAN, I(+)/I(-), F/F(+)/F(-)), readable by aimless/mtz2sca/ANODE.
rugnux: Always write merged reflections as both <prefix>.mtz and <prefix>.cif; the --scaling-output selector and text .hkl output are removed.
rugnux: Add a detector-plane modulation (flat-field) correction surface to rotation scaling (cross-validated, on by default; --no-scaling-corrections disables all), dropping R-meas.
rugnux: Add optional stills detector-plane modulation (--stills-modulation, default off) - the same cross-validated surface for the on-the-fly stills path.
Bragg integration: Local background is now a symmetric trimmed mean of the ring (--background-trim <f>, default 0.10; monochromatic), improving <I/sigma> and resolution-edge CC1/2.

1.0.0-rc.159¶

This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use.

rugnux: Add --model model.pdb - score the merged data against an atomic model and compute initial maps. It reports R-work/R-free (scaling the model to the observed amplitudes with an overall scale, an anisotropic B and a flat bulk solvent - the standard few-parameter model, so a batch of maps stays directly comparable) and writes 2Fo-Fc / Fo-Fc electron-density maps (CCP4) plus a map-coefficient MTZ. The structure itself is not refined; the model is only re-fractionalised into the data cell.
rugnux: The merged reflection output now carries French-Wilson amplitudes (|F| and its sigma) next to the intensities - MTZ F/SIGF, mmCIF _refln.F_meas_au, and the text HKL - computed with the correct centric/acentric Wilson prior and epsilon multiplicity, so a downstream program (e.g. phenix.refine) can refine against amplitudes. The intensity columns are unchanged.
rugnux: R-free test-set flags are now assigned deterministically and consistently across symmetry - a Bijvoet pair I(+)/I(-) is never split between the work and free sets, and the assignment is a reproducible per-hkl hash that depends only on the reflection index, so every dataset of one crystal form gets the same ~5% free set (what a multi-dataset campaign such as PanDDA needs). On small data the fraction is floored so the test set stays large enough for a stable R-free (~500 reflections, capped at 10%); it stays flat at 5% on ordinary data. When a reference MTZ carries a FreeR_flag column its test set is imported instead, letting a whole campaign inherit one shared free set.
rugnux: A reference MTZ (--reference-mtz) can now fix the space group and cell for rotation data too (previously rejected), without being used to scale - the rotation merge stays self-consistent. When the crystal has an indexing (merohedral) ambiguity - a lattice symmetry higher than its Laue symmetry, e.g. P3/P4/P6/C2 - the reference also resolves it: each candidate reindexing (identity plus the twin-law cosets of the metric symmetry) is scored by its intensity correlation against the reference and the data are re-merged in the best-correlating one. This is a metric-preserving relabelling of hkl (the cell is unchanged) and a no-op for a holohedral crystal (which has no twin laws).
rugnux: --model validation now aligns the data to the model before scoring - the observed reflections are reindexed into the model’s enantiomorph when the two differ only by hand (indistinguishable from merged intensities). A merohedral indexing ambiguity is resolved against the reference MTZ when one is given (so a whole campaign shares one indexing convention); only with a model and no reference does validation fall back to fitting each candidate reindexing and keeping the lowest R-free.
rugnux: De-novo symmetry - recover a genuine high-symmetry group whose data are imperfectly scaled. Such a merge’s within-orbit chi² lands just past the self-consistency bound (each real symmetry step adds a little systematic scatter), right where a merohedral twin also lands, so the chi² ratio alone cannot separate them. The candidate is now rescued when the extra intensity-proportional systematic error it invokes stays small relative to the confirmed subgroup - a genuine symmetry step gains multiplicity without inflating the merge error model’s b, whereas a twin forces non-equivalent reflections together and b balloons. Fixes a cubic case (I23 instead of I222) with no change to any other crystal in the test battery, including the twins that must stay in their lower symmetry.
Docs: Document the French-Wilson amplitude estimation, R-free flagging, reference-based space-group/ambiguity resolution, and model-based validation/maps in CPU_DATA_ANALYSIS.md.
Frontend: The status-bar pill now shows a progress bar during detector calibration (previously only during measurement), and the calibration state and its button are labelled “Calibration”/“CALIBRATE” (the internal Pedestal state name is unchanged for back-compatibility).

1.0.0-rc.158¶

This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use.

Analysis: The azimuthal-integration solid-angle correction now follows the incidence angle to the detector normal (cos^3 of that angle) instead of cos^3(2*theta), so it is correct for a tilted detector and matches PyFAI solidAngleArray and MAX IV azint (unchanged for an untilted detector). Crystal geometry refinement (XtalOptimizer) no longer silently ignores an imported PONI rot3 (rotation about the beam): it is applied as a fixed rotation in the residual so refinement stays consistent with the rest of the pipeline. Polarization and azimuthal binning already honoured rot3 through the full PONI rotation.
jfjoch_viewer: Open datasets on the WSL2/UNC filesystem (paths starting \\); write processing outputs next to the input file, with a Browse button and independent _process.h5 / merged .mtz/.cif toggles; and show the determined space group in the merge-statistics window.
jfjoch_viewer: Connect to a broker over https (an http/https selector in the connect dialog), and keep the HTTP connection alive across reads for faster live-follow.
jfjoch_viewer: Time out stalled HTTP requests (5 s) so an unreachable broker cannot hang the reader thread, and drop the cached pixel mask when switching data source.
rugnux: Accept an absolute -o output prefix in offline processing.
rugnux: Faster two-pass rotation indexing - the first pass now runs its FFT indexing and geometry refinement in parallel (results unchanged).
rugnux: Rotation indexing now works on standard DECTRIS datasets that store no spots - the first pass finds spots itself instead of failing.
rugnux: De-novo symmetry robustness - don’t over-promote a merohedral twin to the holohedral group (keep e.g. R3, not R32), make the intensity second-moment twinning statistic robust on weak/mis-integrated data, and don’t flag twinning in holohedral Laue classes where no twin law can exist.
jfjoch_writer: Fold the refined beam centre into the NXmx detector translation vector too (not only the informational beam_center fields), so a reprocessed _process.h5 has a self-consistent refined geometry.
Robustness: Harden size handling of untrusted input in TIFF reading and raw-TCP frames.
Packaging: The self-contained Linux viewer .tgz now bundles cuFFT, so it runs without a system CUDA toolkit (.deb/.rpm are unchanged, distro-managed).
Docs: Documentation updated to match the current analysis code and CLI.

1.0.0-rc.157¶

This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use.

rugnux: Rebrand the offline data-processing subsystem as rugnux and consolidate all offline analysis into the single rugnux binary - jfjoch_process is now rugnux, the former jfjoch_azint is now rugnux --azint-only, and jfjoch_scale is now rugnux --scale (see the new docs/NAMING.md and docs/RUGNUX.md). Scaling and merging are on by default for rotation and stills (--no-merge disables them), replacing the previous opt-in -M, --scale-merge.
rugnux: CLI fixes - default -N to all hardware threads, parse numeric option arguments strictly (reject non-numeric or trailing input instead of silently yielding 0), require --wavelength > 0, and correct the reproduced command line and --scale reference-cell handling.
rugnux: De-novo space-group improvements - recover genuine high symmetry and centred Bravais lattices from intensities, add an automatic CC1/2 high-resolution cutoff, and report L-test twinning statistics.
rugnux: Index weakly-diffracting low-resolution rotation data that previously failed (e.g. F-cubic crystals that diffract only to ~4 A on a detector reaching ~1.5 A). The per-frame indexing gate now measures the indexed fraction only within the resolution range the lattice actually diffracts to, so the many sub-diffraction ice/noise spots no longer make the fraction floor unreachable; the two-pass first pass tries several image-sampling schemes (spread across the whole rotation vs a consecutive wedge whose native stride keeps a reflection’s rocking curve continuous, letting the FFT resolve a long axis) and keeps the one that indexes the most frames; and the de-novo space-group search no longer discards all reflections (and crashes) when every resolution shell falls below <I/sigma> = 1.
rugnux: Lower the low-resolution R-meas for strongly-diffracting rotation data - drop edge-of-sweep truncated fulls whose rocking curve was captured below --min-captured-fraction (default 0.7 for rotation), and report R-meas only over the observations kept by outlier rejection (matching XDS). The 0.7 default also strips the partiality-extrapolated fulls that dominate the intensity second moment on weakly-diffracting crystals, so the de-novo space-group search is no longer starved by the error-model I/sigma floor and recovers the correct symmetry (e.g. for F-centred cubic lattices that would otherwise be under-assigned).
rugnux: Write the refined geometry (beam, tilt, axis) to _process.h5 and place non-standard mmCIF items under a reserved jfjoch prefix.
jfjoch_broker: Ordinary acquisition failures (receiver/writer/analysis problems, missed packets, writer disconnect) now return to the Idle state with an Error-severity message, so a run can be retried without an expensive re-initialisation; only failures that leave the detector in an undefined state (new JFJochCriticalException, e.g. PCIe/FPGA faults) go to the Error state and force re-initialisation.
jfjoch_broker: A synchronous /start now reports its failure to the HTTP caller instead of returning HTTP 200, and an incomplete or truncated dataset (missing packets, writer disconnect) is reported as an error rather than a “reduce frame rate” warning.
jfjoch_broker: Drop uncollected placeholder rows (number = -1) from the scan_result REST endpoint.
jfjoch_broker: Fix the inverted per-image compression ratio reported by the Lite receiver (was compressed/uncompressed instead of uncompressed/compressed).
jfjoch_broker: Bragg integration adds a quantization-noise variance floor with a box-sum fallback, and treats the type-maximum marker as an invalid pixel for unsigned image types.
jfjoch_writer: Detect file-overwrite conflicts at start for back-channel transports, and reset the writer when end-of-collection finalisation fails.
jfjoch_viewer: Preview overlays follow the geometry (resolution/ROI arcs, true beam centre, predictions, coral secondary-lattice spots, legend), add save-as-JPEG, and fix an HTTP live-follow memory leak.
Frontend: Improved aesthetics and usability, and added in-browser pixel-mask and JUNGFRAU-pedestal visualisation.
CI: Name the Windows installer jfjoch-viewer-* instead of jfjoch-*.

1.0.0-rc.156¶

This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use.

jfjoch_process: Major rotation (rot3d) data processing overhaul - robust profile-fit integration, Cauchy-loss scaling with optional absorption surface, de-novo indexing and space-group/centering determination fixes, and merging statistics + ISa in the mmCIF output.
jfjoch_process: Bragg integration now runs on the GPU in the offline/non-FPGA workflow (one box-sum + profile-fit engine, GPU when available, CPU otherwise); the FPGA workflow integrates on the CPU directly from the assembled image. The previous standalone integrators are removed.
jfjoch_process: Deterministic Bragg prediction - when more reflections are predicted than fit the output, they are ranked by distance to the Ewald sphere before truncation, so repeated runs produce identical reflections.
jfjoch_process: Judge systematic absences by resolution-normalised intensity instead of I/sigma alone, so screw axes are no longer missed when the error model under-estimates sigma on weak axial reflections (e.g. the monoclinic 2_1 screw).
jfjoch_process: GPU-accelerated rotation scaling and merging (RotationScaleMerge), substantially faster than the previous CPU path.
jfjoch_process: Unify still and rotation processing on a single –force-still flag (replaces the -P partiality-model option); rotation is auto-detected from the goniometer and processed as rot3d two-pass by default, the default reflection output is mmCIF, and the experimental –reciprocal-profile option is removed.
jfjoch_process: Add EXPERIMENTAL ice-ring detection (–detect-ice-rings) that excludes ice reflections from scaling.
jfjoch_broker: The Bragg integration model (profile-fit Gaussian, empirical, or box-sum) is now selectable via the REST API (/config/bragg_integration) and the web frontend.
jfjoch_broker: Write smargon chi/phi goniometer positions to NXmx; read sensor thickness/material from HDF5 metadata.
jfjoch_writer: Don’t write empty grid-scan position arrays when the dataset has no images.
Compression: Add BSHUF_ZSTD_RLE_HUFF, make compression size-aware (drop frames that don’t fit rather than aborting), and add the jfjoch_recompress tool.
jfjoch_viewer: Report “Multiple lattices detected” and grey out “Analyze dataset” on a live connection.
jfjoch_viewer: Frontend fixes - detector settings widget, panel/preview overflow, and navigation icons.
CI: Build Windows (CUDA and non-CUDA) installers.
CI: Ship jfjoch_viewer to the release as a Linux-agnostic .tgz.

1.0.0-rc.155¶

This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use.

jfjoch_process: Remove pixelrefine option (replaced with ProfileIntegrate2D)
jfjoch_viewer: Some graphical improvements.
jfjoch_viewer: Simplify und unify data analysis settings.
jfjoch_writer: Add TCP keepalive to increase robustness if jfjoch_broker “dies” in the middle of data acquisition.

1.0.0-rc.154¶

This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use.

jfjoch_broker: Fix to TCP file pusher (remove kernel zero copy to improve reliability)

1.0.0-rc.153¶

This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use.

jfjoch_broker: Add EXPERIMENTAL pixelrefine mode for image processing
jfjoch_broker: Allow to load user mask from 8-bit and 16-bit TIFF files
jfjoch_broker: Add ROI calculation in non-FPGA workflow
jfjoch_broker: Fixes to TCP image pusher
jfjoch_broker: Remove NUMA bindings
jfjoch_broker: Improvements to indexing
jfjoch_broker: For PSI EIGER, trimming energies are taken from the detector configuration (now compulsory) instead of hardcoded values
jfjoch_writer: Save ROI definitions and the per-pixel ROI bitmap in the master file; azimuthal ROIs support phi (angular) sectors
jfjoch_viewer: Major redesign with dockable panels and saved layouts, plus on-canvas creation/move/resize of box, circle and azimuthal ROIs
jfjoch_viewer: Run jfjoch_process reprocessing jobs from inside the GUI and overlay per-run results

1.0.0-rc.152¶

jfjoch_broker: Fix bounds for azimuthal integration for Q spacing (allow Q of 1e-5)
jfjoch_viewer: Adjust Q bounds for azimuthal integration
jfjoch_azint: Add tool to do quick azimuthal integration

1.0.0-rc.151¶

jfjoch_broker: For PSI EIGER detector allow to disable individual half-modules by putting empty hostname

1.0.0-rc.150¶

jfjoch_broker: When in FPGA workflow (with PSI detectors) azimuthal integration might be forced to CPU - this will require more computational power, but it enables more integration bins and reports standard deviation of each bin.
jfjoch_broker: Raise error if one is in FPGA flow and there are too many azimuthal integration bins.

1.0.0-rc.149¶

XDS plugin: Fix HDF5 mutex to run on multiple processors

1.0.0-rc.148¶

This is an UNSTABLE release. The release has significant modifications for data processing - in case of troubles go back to 1.0.0-rc.144.

jfjoch_broker: Improve azimuthal integration (add <I^2> calculation)
jfjoch_broker: Fixes around indexing, aiming to handle multi-lattice crystals (work in progress, it is not fully integrated)
jfjoch_writer: Save mean(I), stddev(I), and count(I) for each azimuthal bin

1.0.0-rc.147¶

This is an UNSTABLE release. The release has significant modifications for data processing - in case of troubles go back to 1.0.0-rc.144.

CI pipeline builds software with x86_64-v3 architecture, it should be compatible with practically all x86 hardware manufactured after 2015.
jfjoch_viewer: Add reciprocal space viewer
jfjoch_process: Two pass algorithm that does spot finding/indexing + integration of full dataset
jfjoch_process: Improve logic for rotation indexer, to make execution more deterministic (still work in progress)

1.0.0-rc.146¶

This is an UNSTABLE release. The release has significant modifications for data processing - in case of troubles go back to 1.0.0-rc.144.

jfjoch_broker: Add a lattice-orientation-only refinement option, in addition to full refinement (beam center, lattice orientation, lattice dimension)
jfjoch_process: Generate a dedicated file (_process.h5), which can be used as a replacement for the _master.h5 file for a reanalyzed dataset.
jfjoch_process: Improve the performance of scaling and merging, implement on the fly scaling.
jfjoch_writer: All final data analysis results are repopulated in the _master.h5 file.
jfjoch_scale: Dedicated tool for rescaling/merging existing data.
jfjoch_viewer: Fix bugs where pixel labels where displayed on a wrong pixel.

WARNING! Scaling and merging are experimental at the moment, and may not provide reasonable results for the time being.

1.0.0-rc.145¶

This is an UNSTABLE release. The release has significant modifications for HDF5 writing logic - in case of troubles go back to 1.0.0-rc.144.

Default HDF5 writing mode is with VDS, not soft-links - this improves DIALS compatibility and makes format more future-proof, NXmx legacy format might be phased-out in the future.
XDS plugin: Improve performance of VDS reading.
jfjoch_writer: Significant improvement on how file systems I/O are handled through a dedicated pass-through VFD.
jfjoch_writer: Clean-up of HDF5 routines to better handle issues.