Skip to content

fix(trim): bound per-coordinate trimming so history-heavy columns don't stall - #30

Open
clawd131662[bot] wants to merge 1 commit into
masterfrom
feat/co-trim-bounded
Open

fix(trim): bound per-coordinate trimming so history-heavy columns don't stall#30
clawd131662[bot] wants to merge 1 commit into
masterfrom
feat/co-trim-bounded

Conversation

@clawd131662

@clawd131662 clawd131662 Bot commented Aug 3, 2026

Copy link
Copy Markdown
Contributor

Problem

co:trim's nightly cron had been running for ~23 hours each night while advancing
its checkpoint not at all for ~2 days: db/trim_state.yml stayed frozen while
every run re-scanned the same ~700k-rowid window and then died.

Root cause — to keep a hot coordinate's newest --keep rows, co:trim ran:

Block.where(wid:, x:, y:, z:, action:).where(rowid: ..upper).order(rowid: :desc).pluck(:rowid)

The (wid, x, z, time) index can't satisfy ORDER BY rowid, so for a coordinate
that had quietly accumulated millions of rows (an end-portal-arrival machine column,
~2.1M rows) MariaDB sorted the coordinate's entire history in a single statement. On
the server's HDD that pluck ran > 3600s, hit max_statement_time, and raised
StatementTimeout. The exception was caught by trim's rescue before
save_trim_state
, so the segment checkpoint was never written — every following
night restarted from the same rowid and repeated the doomed pluck, trimming nothing.

Fix

  • Locate the cutoff via ORDER BY time DESC (the index's own order):
    offset(keep-1).limit(1).pick(:rowid) costs O(keep) look-ups instead of sorting the
    whole history. Measured on the exact coordinate that was timing out: 0.118s
    (was > 3600s).
  • Delete everything below the cutoff in bounded --step slices
    (limit(step).pluckdelete_all by rowid), so no single statement scales with the
    backlog.
  • Isolate per-coordinate failures: a StatementInvalid on one coordinate is logged
    (trim_coordinate_skipped_message) and skipped instead of aborting the whole segment
    and forfeiting the checkpoint.

Semantics are unchanged — still "keep the newest --keep rows per
(wid, x, y, z, action)". The cutoff is now chosen by time rather than rowid; both
are monotonic with insertion, so the kept set matches in practice (the same approach as
the host's proven trim_grid.sh cleanup script).

Validation

This repo has no automated suite, so validated manually against the production DB:

  • ruby -c and rubocop clean (only the pre-existing Metrics/ClassLength offense).
  • New cutoff query on the offending coordinate: 0.118s.
  • co:trim --dry-run over a live window: correct output, victims = total − keep on
    every line, no runtime errors.

Follow-ups (not in this PR)

  • Adaptive throttle / consumer-lag health gate, so a first-time multi-million-row flush
    never stresses CoreProtect's single-threaded online consumer (routine daily churn is
    trivial, but a newly-discovered mega-column currently deletes back-to-back).
  • Docs (README / CLAUDE.md) still use the old "pluck" wording.

🤖 Generated with Claude Code

…'t stall

co:trim located each hot coordinate's keep-th newest row with `ORDER BY rowid
DESC`, which the (wid, x, z, time) index can't satisfy, so a column with millions
of rows sorted its whole history in one statement and blew past max_statement_time.
The raised StatementTimeout aborted the run before save_trim_state, so the
checkpoint never advanced and every daily cron re-scanned the same window for
~23h, trimming nothing.

- Locate the cutoff via `ORDER BY time DESC` (the index's own order), costing
  O(keep) look-ups instead of sorting the coordinate's entire history.
- Delete everything below the cutoff in bounded `--step` slices, so no single
  statement scales with the backlog.
- Isolate per-coordinate failures: a StatementInvalid on one coordinate is logged
  and skipped instead of aborting the segment and forfeiting the checkpoint.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant