Skip to content

Add auto-alignment for HDR bracket merging#21420

Open
da-phil wants to merge 5 commits into
darktable-org:masterfrom
da-phil:add_hdr_merge_auto_align
Open

Add auto-alignment for HDR bracket merging#21420
da-phil wants to merge 5 commits into
darktable-org:masterfrom
da-phil:add_hdr_merge_auto_align

Conversation

@da-phil

@da-phil da-phil commented Jun 25, 2026

Copy link
Copy Markdown
Contributor

Implements automatic image registration in the lighttable merge HDR path so handheld/tripod brackets can be merged without ghosting, via OpenCV.

Auto-alignment processing steps:

  • Each bracket frame is collapsed from its raw CFA mosaic to a CFA-free luma proxy (~0.625× resolution) by averaging stride-1 2×2 windows, then percentile-normalised and gamma-encoded to an 8-bit image.
  • SIFT keypoints are detected on the 8-bit proxy, filtered to a 6 px scale floor, then spatially balanced to 5000 per frame across a 6×6 grid so neither frame dominates the matcher.
  • Descriptors are matched with FLANN kNN, filtered by Lowe ratio (0.75) and mutual-NN consistency, then spatially subsampled to 1800 matches before RANSAC to avoid bias toward dense bright regions.
  • RANSAC estimates an 8-DOF homography; if it yields fewer than 50 inliers or below 40 % inlier ratio, estimateAffine2D is tried as a fallback. If inliers cluster into ≤ 2 grid cells the model is further degraded to a pure translation to avoid wild extrapolation.
  • The result is sanity-checked (translation < 30 % of the image diagonal, scale within 0.5–2×); frames that fail are accumulated unwarped, so the output is never worse than today's alignment-free merge.
  • The homography is rescaled from proxy to full-sensor coordinates and applied with a CFA-aware resampler that only reads same-color photosites, preserving the Bayer/X-Trans mosaic phase so the warped frame drops directly into the existing weighted accumulator.

A more detailed design overview can be found here:
https://github.com/da-phil/darktable/blob/a5a803c5447894faa208a9507a5c82e5fb38e260/dev-doc/HDR_Alignment_Design.md

Key changes

  • src/common/hdr_alignment.{h,c}: pure-C layer — CFA luma proxy build, percentile normalisation, proxy→full-res homography rescale, CFA-aware same-color warp (Bayer exact, X-Trans approx), reliability/sanity gate, per-frame orchestration
  • src/common/hdr_alignment_cv.cc: OpenCV C++ backend — SIFT detect, FLANN kNN + Lowe ratio + mutual-NN, spatial keypoint balancing (5000-each), findHomography RANSAC, affine fallback, cluster- degradation to translation-only, spatial match subsampling
  • control_jobs.c: wire alignment state into dt_control_merge_hdr_t; set_reference on first frame, align_frame on each subsequent frame
  • Optional USE_OPENCV CMake flag (default ON); without OpenCV the feature compiles to a no-op and the merge is unchanged
  • Seven runtime prefs in darktableconfig.xml.in (auto_align, auto_reference, proxy_scale, feature_gamma, clahe_clip, sift_keypoints, debug_images)
  • DT_DEBUG_HDR_MERGE channel (-d hdr_merge), debug image dump
  • cmocka unit tests: synthetic homography round-trip + probe ranking
  • CI/CD: add OpenCV per-module packages to Dockerfile, Brewfile, ci.yml, nightly.yml; README + Windows packaging docs updated

App images can be found here:
https://github.com/da-phil/darktable/actions/runs/28748026727#artifacts

Disclaimer: this work has been co-created with Claude.

Fixes: #17326

@wpferguson

wpferguson commented Jun 25, 2026

Copy link
Copy Markdown
Member

Are the settings in preferences pretty much set and forget (don't need to be messed with) or useful to be able to adjust without having to go to preferences?

I created an HDR, then adjusted the settings and tried to create another HDR using the same images as the previous. The second HDR overwrote the first HDR. Probably should use the on conflict naming.

EDIT: In my very quick test, the images looked fine. No alignment errors were observed.

@da-phil

da-phil commented Jun 26, 2026

Copy link
Copy Markdown
Contributor Author

Are the settings in preferences pretty much set and forget (don't need to be messed with) or useful to be able to adjust without having to go to preferences?

I believe some of them might be removed after more testing, or we remove them from the GUI and keep them as expert settings in dartablerc only.

The second HDR overwrote the first HDR. Probably should use the on conflict naming.

This behaviour exists since day 1 of the HDR merge feature and is not scope of this PR. I plan to improving the HDR merge feature after this PR, as I discovered a lot of bad behaviours with the actual HDR merge functionality. I want to make it closer to the hdrmerge algorithm, which - by the way - was also recently adopted in vkdt. It prevents ghosting much better as the naive merge algorithm in darktable.
Then we can also talk about other improvements of the actual HDR merge behaviour, such as how do we deal with existing merged DNG files.

EDIT: In my very quick test, the images looked fine. No alignment errors were observed.

Perfect! Can you try some really challenging handheld exposure brackets?

@ralfbrown ralfbrown added feature: new new features to add scope: image processing correcting pixels labels Jun 26, 2026
@wpferguson

Copy link
Copy Markdown
Member

This behaviour exists since day 1 of the HDR merge feature

At that time there was only 1 possible output. This PR introduces settings, making multiple outputs possible. Since settings are available, users will try and tune the HDR output to find the best settings. You could probably steal the conflict code from the exporter or just do a simple test if the file exists and then append _xx to the filename where xx is a 2 digit sequence number.

Can you try some really challenging handheld exposure brackets?

Everything I have can be aligned by simple x y alignments.

@da-phil da-phil force-pushed the add_hdr_merge_auto_align branch from a5a803c to 938a8f3 Compare June 27, 2026 14:07
@da-phil

da-phil commented Jun 27, 2026

Copy link
Copy Markdown
Contributor Author

At that time there was only 1 possible output. This PR introduces settings, making multiple outputs possible. Since settings are available, users will try and tune the HDR output to find the best settings. You could probably steal the conflict code from the exporter or just do a simple test if the file exists and then append _xx to the filename where xx is a 2 digit sequence number.

I prefer to do this in another PR, I don't want to change too many things at a time.
This PR is only about auto-alginment for the HDR merge functionality and getting feedback from folks who requested the feature.

EDIT: Okay, the more I think about this the more sense it makes, especially for testing the feature with the ability to create multiple new HDR DNG files with different settings will be much easier with that filename numbering schema. If we settle on a specific set of parameter values, we can also remove this again.
Will push another commit soon.

@da-phil

da-phil commented Jun 28, 2026

Copy link
Copy Markdown
Contributor Author

FYI: I did not test the auto-alignment with X-Trans (Fuji) RAW files. It would be great if anybody could cover this case or provide exposure brackets for testing. Thanks!

@da-phil da-phil marked this pull request as ready for review June 29, 2026 15:14
da-phil added 5 commits July 4, 2026 22:31
Implements automatic image registration in the lighttable merge HDR
path so handheld/tripod brackets can be merged without ghosting, via OpenCV.

Key pieces:
- src/common/hdr_alignment.{h,c}: pure-C layer — CFA luma proxy
  build, percentile normalisation, proxy→full-res homography rescale,
  CFA-aware same-color warp (Bayer exact, X-Trans approx),
  reliability/sanity gate, per-frame orchestration
- src/common/hdr_alignment_cv.cc: OpenCV C++ backend — SIFT detect,
  FLANN kNN + Lowe ratio + mutual-NN, spatial keypoint balancing
  (5000-each), findHomography RANSAC, affine fallback, cluster-
  degradation to translation-only, spatial match subsampling
- control_jobs.c: wire alignment state into dt_control_merge_hdr_t;
  set_reference on first frame, align_frame on each subsequent frame
- Optional USE_OPENCV CMake flag (default ON); without OpenCV the
  feature compiles to a no-op and the merge is unchanged
- Seven runtime prefs in darktableconfig.xml.in (auto_align,
  auto_reference, proxy_scale, feature_gamma, clahe_clip,
  sift_keypoints, debug_images)
- DT_DEBUG_HDR_MERGE channel (-d hdr_merge), debug image dump
- cmocka unit tests: synthetic homography round-trip + probe ranking
- CI/CD: add OpenCV per-module packages to Dockerfile, Brewfile,
  ci.yml, nightly.yml; README + Windows packaging docs updated
Sync documentation with implementation again.

Software quality:
- control_jobs: restore the authoritative per-frame decoded-buffer guard
  (filters/channels/datatype) that the flags-only pre-check cannot replace
  (monochrome raws carry DT_IMAGE_RAW yet decode to filters==0; HDR/mono flags
  can lag the decode). Also reject monochrome in the up-front validator.
- hdr_alignment: the X-Trans degenerate resample fallback now searches outward
  for the nearest same-colour photosite instead of returning any colour, so a
  warped cell never gets a wrong-colour sample (preserves mosaic phase).
- Remove the dead motion-model parameter/enum from dt_hdr_alignment_new().
- Auto-reference probe pre-pass is now cancellable and advances the progress
  bar (was uninterruptible and looked hung on large brackets).
- Report the cluster-degraded translation model accurately in the metrics log
  (new used_translation stat) instead of mislabelling it "homography".
- _resolve_debug_dir creates the DT_HDR_DEBUG_IMAGE_DIR override if missing;
  dt_hdr_cv_count_features drops a no-op CLAHE call and a defensive clone;
  rename the moving-image cv::Mat i -> mov_img; write the reference debug
  visuals once instead of per frame.

Tests:
- test_hdr_alignment: CFA-modulated Bayer alignment test (per-channel gains, so
  a colour-blind warp would introduce crosstalk) — the headline property.
- test_hdr_alignment_internal: X-Trans same-colour isolation test exercising the
  border fallback (sampling colour c only ever reads colour-c photosites).
- Cache the reference FLANN index. The reference (template) descriptors are
  fixed for the whole merge, but the image->template KD-tree was rebuilt for
  every one of the N-1 moving frames. Train it once in
  dt_hdr_cv_features_create() and store it on RefFeatures; only the
  template->image direction still builds a per-frame index over the moving
  descriptors. Match semantics (queryIdx/trainIdx) are unchanged.

- Auto-reference probe builds its proxy at the probe resolution directly. SIFT
  in the probe runs at <= AUTO_REFERENCE_PROBE_DIM (1500px) anyway, so building
  the full DT_HDR_PROXY_SCALE proxy and letting OpenCV downscale it wasted work:
  for a 24-100MP raw the intermediate proxy was 2.5-4.9x larger than needed.
  Now the proxy is built (and percentile-normalized) directly at probe scale;
  the ranking stays self-consistent because all frames share dimensions.

- Cap the FLANN parallel-sections team to num_threads(2): only two sections do
  work, so there is no reason to wake the full OMP team.

- Skip the redundant full-frame copy for a static frame. When the warp is a
  reliable no-op (sub-pixel corner motion) or identity, align_frame now returns
  FALSE and leaves `out` untouched, so the caller accumulates its own source
  buffer instead of a byte-identical copy (~w*h*4 bytes saved per static frame).
  The header contract is updated: `out` is only guaranteed populated on TRUE.

- Free the moving float proxy right after the u8 proxy is built (it is only an
  intermediate), trimming peak footprint during the SIFT stage.

Not done: the affine/translation warp fast path (incremental source-coordinate
stepping when H[6]==H[7]==0). It is gated behind collapse(2)/parallelization
changes for a minority of per-pixel warp cost while SIFT dominates wall time, so
it is left as profile-gated backlog per the review.
@da-phil da-phil force-pushed the add_hdr_merge_auto_align branch 3 times, most recently from b88115d to 927c845 Compare July 5, 2026 16:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

feature: new new features to add scope: image processing correcting pixels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Better HDR merge with auto-allignment

3 participants