fix(gateway): continuously drain retention cleanup - #467
Merged
Conversation
Run bounded maintenance every minute on Cloudflare and Node so hot dump keys can outpace production ingestion without concentrating work in an hourly burst.\n\nKeep the existing durable queue and fairness semantics, reduce dump batches to 50 rows, and derive the spilled-file collector budget from the maximum files a tick can retire. Add regression coverage for cadence, bounded catch-up under sustained input, and the collector cap.\n\nVerified with the full repository verify pipeline.
Protect the per-minute maintenance budget with a D1-backed lease so overlapping Node timers, duplicate cron delivery, or concurrent Worker isolates cannot multiply cleanup work while the database is slow.\n\nKeep deployment cadence owned by each platform, validate stale-lease takeover, and exercise the real four-key dump-to-file collection path for the derived 400-file collector bound.
Move the global lease into a dedicated repository, renew it between maintenance phases, and shorten stale recovery to five minutes so failed cleanup cannot suppress an hour of ticks.\n\nExtract the Node timer wiring into a side-effect-free module and verify the startup and one-minute schedules directly.
Renew the global lease every minute while a bounded phase is awaiting I/O, and stop before the next side-effecting phase if ownership cannot be confirmed.\n\nCover long-running and failed-heartbeat paths, pin the Node entry wiring, and move repository and orchestration tests to their owning test modules.
Unreference the cross-runtime heartbeat on Node, surface renewal failure from the final phase, and retain concurrent heartbeat and release errors without masking the original failure.\n\nMake the Node production scheduler itself injectable and behaviorally tested, and narrow the cron rationale to the bounded batch invariant.
Exercise the runner's five-minute abandoned-lease takeover and the Node scheduler's production default callback and timers so the recovery window and platform composition cannot silently drift.
Remove the deleted hourly cron referent from the Wrangler comment and describe only the current minute cadence and bounded batch invariant.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
At the configured cadence, one persistently hot key has 72,000 dump rows/day of cleanup capacity while every deletion and collection statement remains bounded.
Test Plan
pnpm run verify— 543 test files and 5,644 tests passed.