Skip to content

feat(motiondiff0r): add temporal frame differencing filter - #320

Merged
jaromil merged 7 commits into
dyne:masterfrom
phoebus-84:claude/trusting-turing-8v5zc7
Nov 5, 2026
Merged

jaromil merged 7 commits into
dyne:masterfrom
phoebus-84:claude/trusting-turing-8v5zc7

Conversation

@phoebus-84

@phoebus-84 phoebus-84 commented Nov 4, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Adds motiondiff0r, a new filter that reveals motion by comparing the current frame with a delayed copy of the stream. The delayed frame is inverted and laid over the current one: everything that stays still cancels out to neutral gray, while moving or changing regions show up as light and dark (positive / negative) shapes.

The idea comes from this video, which shows the trick of duplicating footage, inverting one copy, overlaying it and offsetting it in time: https://www.youtube.com/watch?v=NSS6yAMZF78

Modes

Mode Value Result
Negative overlay 0 (1 - blend) * F(t) + blend * (255 - F(t - dt)); with blend 0.5 static areas become mid gray
Absolute difference 0.5 abs(F(t) - F(t - dt)); static areas become black
Motion mask 1 current frame with the amount of change written to the alpha channel, for compositing with other filters

Parameters

Name Type Meaning
Time Offset double 0..1 maps to 0..2 seconds; 0 compares with the previous frame
Mode double see above
Blend double opacity of the inverted delayed frame (overlay mode)
Contrast double 0..1 maps to 1x..16x amplification of the differences
Threshold double ignore differences below this, to suppress noise
Monochrome bool compare luminance only, grayscale result

Implementation notes

  • The delayed frame is picked from a short history using the host timestamps, so the offset is independent of the frame rate and works with variable frame rates.
  • Frames at or after the current time are dropped from the history, so seeking backwards or rendering the same time again behaves correctly.
  • History uses a fixed ring allocated during construction: at most 128 frames and 1 GiB of pixel storage per instance, including the current frame. This fits 128 frames at 1080p or 32 at 4K. The effective offset is shortened when the requested history does not fit; dimensions that cannot fit two frames are rejected. Rendering does not allocate buffers.
  • Released pixel slots are reused before unused slots, so short offsets keep a small working set. At 1080p with zero offset, the measured resident-memory increase stays about 16 MiB through 200 frames. The full bounded reservation remains; memory touched by a larger history may remain resident afterward.
  • The input is copied before rendering, so in-place rendering (in == out) is supported.
  • C++ using frei0r.hpp, RGBA8888, no new dependencies.

Testing

  • New test/motiondiff0r-contract.c (registered in ctest) covering the three modes, frame selection by timestamp, threshold, seeking backwards, same-time and in-place rendering, and a long run at the maximum offset.
  • All four focused motiondiff0r checks passed under Clang, GCC, and AddressSanitizer with the 1 GiB cap; a module built with exceptions disabled also passed the rendering contract.
  • test/motiondiff0r-memory-contract.cpp verifies byte/frame limits (including the exact 1 GiB two-frame boundary), constructor allocation failures, existing-instance metadata after failure, dense timestamp histories, rendering with heap allocations disabled, and reuse of only two buffers at zero offset or five at 125 ms / 32 fps across wraparound, seeks, and repeated timestamps.
  • cpplint with the CI filters: no findings.
  • Tried by hand on real footage with ffplay -vf "frei0r=motiondiff0r:0.1|0|0.5|0|0|n" on macOS.

@github-actions

github-actions Bot commented Nov 4, 2026 •

Copy link
Copy Markdown

✅ Conventional Commits check passed

All commit messages in this PR now follow the Conventional Commits specification. 🎉

phoebus-84 and others added 2 commits November 4, 2026 17:23
Compares the current frame with a delayed one taken from a short,
timestamp-based history (0-2 s, 0 = previous frame). Three modes:
negative overlay (inverted delayed frame blended over the current one,
static areas turn mid gray), absolute difference (static areas black)
and motion mask (amount of change written to alpha). Parameters for
blend, contrast, noise threshold and luminance-only comparison.
In-place rendering is supported.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XnoxvoRbSLkQ3sTCkxgiyi
Checks the three modes, frame selection by timestamp, threshold,
seeking backwards, same-time and in-place rendering, and a long run
at the maximum offset.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XnoxvoRbSLkQ3sTCkxgiyi
@phoebus-84
phoebus-84 force-pushed the claude/trusting-turing-8v5zc7 branch from 0b56aa9 to 49fa9fa Compare November 4, 2026 17:24
@jaromil

jaromil commented Nov 4, 2026

Copy link
Copy Markdown
Member

nifty old trick :^) I had it made in assembler inside early freej versions, called absdiff, works well when combined with other filters.

@jaromil jaromil left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed all five changed files at 49fa9fa. The existing Test and Website workflows are green. A local Clang 19 core-bundle build succeeds, and all 185 selected native CTests pass, including motiondiff0r-contract and the bundle symbol audit. Optional OpenCV/Cairo/GAVL plugins and the two profiles requiring their development packages were excluded locally.

I found one blocking resource-handling defect, detailed inline. Failure injection against the real shared module makes a frame allocation throw through f0r_update, then the next update segfaults even after allocations recover. The existing tests exercise normal allocation paths but not this failure/recovery path. No implementation changes have been made.

Comment on lines +112 to +119
history.push_back(Frame());
Frame& f = history.back();
f.time = time;
if (!pool.empty()) {
f.pixels.swap(pool.back());
pool.pop_back();
}
f.pixels.assign(in, in + n);

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P1] Make frame allocation failure exception-safe and recoverable

The Frame is inserted into history before its pixel storage is allocated. If pixels.assign throws std::bad_alloc, the exception escapes f0r_update into the C host, and the new history entry remains present with an empty pixels vector. Even a host that catches the exception cannot safely continue: the next update may use that empty entry as history.front(), and process dereferences an invalid delayed-frame pointer.

I reproduced this against the built shared module using valid 8×8 frames: inject failure of the first 256-byte frame allocation, catch bad_alloc around f0r_update at t=0, restore normal allocations, then call f0r_update at t=1/30. The first output is left untouched; the second update terminates with SIGSEGV. No actual memory exhaustion is needed for the reproduction. Frame history makes allocation pressure realistic at larger resolutions and offsets.

Please allocate/populate a candidate frame before committing it to history, handle allocation failures from the history/pool operations as well, and produce defined fallback output without letting exceptions cross the plugin API. Add a regression that fails an allocation and then renders another frame successfully.

phoebus-84 and others added 3 commits November 4, 2026 18:07
Copy the incoming frame into a local buffer before touching the history,
so a failed allocation can no longer leave an empty entry behind that a
later update dereferences. Recycling keeps a single spare buffer and no
longer allocates, and update() catches every exception so none crosses
the plugin API: when memory runs out the frame is compared with itself,
which gives the neutral "no motion" output, and the next frames render
from the intact history.

Add a regression test that replaces the global operator new to fail the
frame copy (first frame, with history, in place) and checks the fallback
output and the recovery. Without the fix it reproduces the reported
segfault.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XnoxvoRbSLkQ3sTCkxgiyi
Limit each instance to 128 frame slots and 128 MiB of pixel storage, allocated during construction. Reuse a timestamped ring without allocating during update, and shorten the effective offset when history does not fit.

Allow the C++ helper to reject failed constructor resources, return NULL on allocation failure, and restore parameter metadata after failed construction. Add memory-budget and allocation-free-rendering coverage, and adapt the existing allocation regression to constructor preallocation.

Verified 187 selected native tests, four focused GCC and ASan checks, and a no-exceptions module build with its rendering contract.
@phoebus-84

Copy link
Copy Markdown
Collaborator Author

Reviewed all five changed files at 49fa9fa. The existing Test and Website workflows are green. A local Clang 19 core-bundle build succeeds, and all 185 selected native CTests pass, including motiondiff0r-contract and the bundle symbol audit. Optional OpenCV/Cairo/GAVL plugins and the two profiles requiring their development packages were excluded locally.

I found one blocking resource-handling defect, detailed inline. Failure injection against the real shared module makes a frame allocation throw through f0r_update, then the next update segfaults even after allocations recover. The existing tests exercise normal allocation paths but not this failure/recovery path. No implementation changes have been made.

Thanks for the review and for pushing 937ebcd — preallocating the history and keeping update() allocation-free is a cleaner fix than my try/catch fallback, which it supersedes. I pulled it and rebuilt locally: the full suite passes (194/194), and motiondiff0r-alloc and motiondiff0r-memory-contract are also clean under ASan/UBSan.

One note for users: with the 128 MiB cap the reachable offset depends on resolution (about 1.4 s at 720p, 0.6 s at 1080p, 0.12 s at 4K at 25 fps). That’s fine for the motion-extraction use case, which mostly uses short offsets.

Would it make sense to size the ring from the selected offset (still capped at 128 MiB), so small offsets don’t reserve the full budget up front? Happy to look into it if you think it’s worth it.

Keep the bounded constructor allocation while decoupling chronological history entries from physical pixel slots. Reuse released slots first so short offsets do not cycle through and touch the entire reservation.

Add working-set regression coverage for zero and positive offsets, history wraparound, seeks, and repeated timestamps.
Update the parameter description and allocation contract, including acceptance at the two-frame boundary and rejection just beyond it. Keep the 128-frame limit and reuse of released buffers.
@jaromil
jaromil merged commit 6bfdebc into dyne:master Nov 5, 2026
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants