Repository navigation
fix(auto-routing-benchmark): isolate dead-lettered lanes so one model cannot wedge a run - #5135
Merged
Merged
Conversation
… cannot wedge a run A benchmark queue message that exhausted its 6 retries dead-lettered silently: its lane's chunk chain was gone, finalizeRunIfComplete waited forever for rows that can never exist, and the 6h stale sweep then failed the whole run — including every cleanly measured model. Consume the DLQ in the worker itself. Each dead message is recorded in the new run_lane_failures table and finalization becomes lane-aware: a run completes once every (model, variant, rep) lane has either all its case rows or a recorded lane death. - Profile runs complete per-entry: finished entries go ready, entries with a dead lane go failed (retry re-admits them, quota-charged). - Platform runs fail fast with the dead lane named; publishing a partial routing table would silently drop a candidate from routing decisions. - The 6h stale sweep stays as backstop for deaths that never reach the DLQ.
iscekic
enabled auto-merge (squash)
August 7, 2026 12:56
iscekic
disabled auto-merge
August 7, 2026 12:56
![kilo-code-bot[bot]](https://avatars.githubusercontent.com/in/2193792?s=60&v=4){kind=small}
Contributor
Code Review SummaryStatus: No Issues Found | Recommendation: Merge Executive SummaryIncremental review of the three follow-up commits since Files Reviewed (7 files)
Previous Review Summary (commit ed0198a)Current summary above is authoritative. Previous snapshots are kept for context only. Previous review (commit ed0198a)Status: 3 Issues Found | Recommendation: Address before merge Executive SummaryThe DLQ consumer's catch-and-always-ack handler defeats its own Overview
Issue Details (click to expand)WARNING
SUGGESTION
Files Reviewed (13 files)
Reviewed by kimi-k3 · Input: 42.5K · Output: 10.3K · Cached: 313.6K Review guidance: REVIEW.md from base branch |
RSO
approved these changes
Aug 7, 2026
…mpt finalization The swallowed error + unconditional ack made the DLQ consumer's max_retries ineffective: if recordLaneFailure committed but finalizeRunIfComplete threw transiently, nothing re-attempted finalization and the run wedged until the 6h stale sweep failed it wholesale — the exact failure mode this PR removes. processDeadLetter now propagates transient errors (same ack contract as the jobs branch); a message still failing after max_retries is dropped and the stale sweep remains the backstop.
…dentity Space-joined keys collide when a model or variant contains a space (the pool schema allows any trimmed non-empty variant). Match the codebase's collision-safe key convention with JSON.stringify tuples, with a regression test where space-joined keys would collapse two entries.
…fail statement builder With entries = [], or(...[]) evaluates to undefined and the WHERE degrades to run_id + status='running', which would fail every running entry of the run. The wrapper already early-returns; the exported builder now refuses the destructive form so a future direct caller cannot produce it silently.
jeanduplessis
approved these changes
Aug 7, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Today a decider run wedges when a queue message exhausts its 6 retries and dead-letters: the lane's chunk chain is gone,
finalizeRunIfCompletewaits forever for rows that can never exist, and the 6h stale sweep then fails the entire run — including every cleanly measured model. Observed in prod: profile runprofile-2026-08-07T09-24-27-661Zstuck for hours with 32/33 entries fully measured; onlycohere/north-mini-code:freedead-lettered.The worker now consumes
auto-routing-benchmark-dlqitself. Each dead message is recorded in a newrun_lane_failurestable, and run finalization becomes lane-aware: a run completes once every (model, variant, rep) lane has either all its case rows or a recorded lane death.ready; entries with a dead lane gofailed(owner Retry re-admits, quota-charged). One failing model no longer fails the other owners' entries.Verification
Visual Changes
N/A
Reviewer Notes
0008_easy_doctor_faustus.sqlviapredeploy.