Skip to content

Auto-Align Layers: match overlapping photos to the active layer by their content - #325

Open
MyronKoch wants to merge 1 commit into
robbietilton:mainfrom
MyronKoch:feat/auto-align-layers
Open

MyronKoch wants to merge 1 commit into
robbietilton:mainfrom
MyronKoch:feat/auto-align-layers

Conversation

@MyronKoch

Copy link
Copy Markdown
Contributor

For #316, the alignment half: no blending or stitching into one layer.

What it does

  • Select two or more pixel layers and choose Layer › Align › Auto-Align Layers, or Auto-Align in the Move bar's Align panel.
  • Each layer is lined up over the active layer, which stays where it is.
  • A match that only shifts a layer moves it by whole pixels, the same as Align does, so a tilt-shift or tripod series isn't resampled.
  • Any other match distorts the layer into place through the existing distort(at:). That was private in Distort.swift and is now internal; nothing else in it changes.
  • It's one undo step and runs off the main thread.
  • Folders, adjustments, live text, shapes and GIF layers aren't included.
  • Layers nothing matches are named in a "Couldn't align" alert and left where they were.

How it matches (AutoAlign.swift)

Vision alone wasn't enough. On two real 1000 px frames that overlap by 45%, VNHomographicImageRegistrationRequest was far off, and VNTranslationalImageRegistrationRequest returned (-186, 728) against a true offset of about (550, 30). Both of them answer with full confidence for a flat gray image too. Vision also registers images of the same size only. So the match is built in three steps:

  1. Phase correlation (Accelerate's 2D FFT, about 512 px) finds the offset, however small the overlap. The edges are tapered over the outer 10% only, because panorama frames overlap at their edges.
  2. Vision then runs on just the overlapping crops, which are the same size, where it works well. It gives a translation and a perspective.
  3. Each candidate is scored by the correlation of brightness where the pixels overlap, which holds across exposures. The best is kept if it scores at least 0.5.
    • A perspective is used only when it covers at least a third of the layer and beats the best shift by more than 0.03. A homography fitted to a narrow overlap matches there but goes wrong beyond it: at 25% overlap it shrank a 400 px frame to 340 px.

Each layer's own pixels are compared, not the layers drawn onto the canvas, because their empty surroundings would look like an edge to match. The result is mapped through both layers' placements, flips included. On the real frames the whole match takes about 0.15 s.

Testing

  • 6 tests in AutoAlignTests:
    • A shifted copy moves back with the same CGImage, so no resampling.
    • Two frames of one scene that share only a quarter of their width land exactly 300 px apart.
    • A copy warped in perspective is distorted back to within a couple of pixels.
    • A flat gray layer is reported and stays put.
    • Only plain pixel layers take part, and the shift-or-distort rule is tested directly.
  • Sabotage checks:
    • With phase correlation disabled, the quarter-overlap test fails (exit 65).
    • With the agreement gate disabled, the flat-gray test fails (exit 65).
  • Full CompositorTests on macOS 27: 886 pass and 1 fails, the ExternalChangeTests file-event timeout (about 106 s), as on Perspective Crop: the Crop tool's Perspective switch straightens what four corners enclose #324. The six ExternalChangeTests pass on their own.
  • Release build, driven by script:
    • Opened a project with two overlapping 1000 × 1000 crops of a photo. One was 550 px to the side and tilted in perspective, set to Difference blend.
    • Selected both layers and chose Layer › Align › Auto-Align Layers.
    • The overlap went black, so the frames match, and Edit showed "Undo Auto-Align Layers".
    • An earlier build that used Vision alone refused the same pair. That is what led to the design above.

🤖 Generated with Claude Code

https://claude.ai/code/session_01WC9aF7W8inTF7p9LZbNXRe

…eir content

Layer > Align > Auto-Align Layers, and Auto-Align in the Move bar's Align
panel, take two or more selected pixel layers and line each one up over
the active layer. A match that only shifts a layer moves it by whole
pixels, as Align does, so a tilt-shift or tripod series isn't
resampled; any other match distorts the layer into place, as a
Cmd-drag distortion does. One undo step.

Phase correlation finds how far apart two frames are, however little
they overlap, and Vision then refines the match over just the part they
share. Vision on its own misses frames that overlap by half, and answers
with full confidence even for a flat gray, so each candidate is checked
against the pixels (the correlation of their brightness, which holds
across exposures) and the best is kept. A perspective is used only when
it shares a third of the layer and clearly beats a shift; one fitted to
a narrow overlap goes astray beyond it. Layers nothing matches are named
in an alert and left where they were.

Refs robbietilton#316.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant