Conversation
The orphan-leave sweep was reachable for exactly one epoch per orphan, so an orphan that survived a boundary became permanently unsheddable. Being orphaned is what expires an allocation: no worker bound means no proofs, so it misses its per-epoch re-confirm at the next boundary and `AllocationBuckets::from_allocations` moves it out of `active` into `expired_epoch`. The sweep drew its candidates from `active` alone, so from that moment nothing could propose a Leave for it — not the orphan path, and not `plan_leaves`, which scores `allocated_descriptors`, itself filtered by `active`. The node then holds allocations it cannot serve, indefinitely, with no automatic way out. Observed on a node with 35 allocations against 15 workers: the 20 unbound ones all read `re-confirm!` and no Leave was ever proposed. Draw orphan candidates from `active` UNION `expired_epoch`, in a `orphaned_allocation_filters` helper. `expired_epoch` deliberately does not join `active` itself — that set is coverage accounting, and an allocation that has not re-confirmed genuinely does not count — so this widens only who may be shed. The existing exclusions still apply, on the expired path as much as the active one: a filter with a worker bound (including an in-flight join), an operator-pinned filter, and one already mid-Leave are all left alone. The leave block's own `!active.is_empty()` guard is widened the same way, so a prover whose every allocation expired still sweeps. An unbound allocation stays on-chain Active, so a Leave is valid for it; only the derived status reads expired. Not changed: the per-epoch re-confirm still queues every `expired_epoch` filter, so an orphan is briefly both re-confirmed and proposed for Leave. Re-confirming re-encodes replicas from local storage, and whether that is reachable without a bound worker is not established here, so narrowing it is left out of this fix. The Leave lands within an epoch, after which the allocation is `Leaving` and out of `expired_epoch`. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011K67csMrXEr4ovegwy2Fmq
A prover holding more allocations than workers shed the wrong shards. Both capacity paths picked orphans — Active allocations with no worker bound — first and score-blind, on the reasoning that "no worker does the work, so leaving them costs nothing." It costs the difference in score. Which allocations hold a worker is decided by bind order in the allocator, not by value: a crashed worker, a reduced core count, or a join batch that outran the idle pool all leave whatever they held bound and push the rest out in arrival order. `WorkerAllocator::rebind_surplus_by_priority` exists to correct that, moving workers off the worst-ranked bound allocations onto the best-ranked orphans. It only promotes an orphan that is still steady Active/Paused, and a leave marks the allocation Leaving — so the score-blind rule raced ahead of the score-aware one and always won. The allocator's own doc comment already stated the contract that was being broken: "the surplus-leave path sheds them, lowest-scoring first — the same order used here, so the shards left unbound are the ones it will propose leaving." Observed on mainnet: 27 allocations against 15 workers. Of the 14 shed, four were on the best-paying ring the node held while five bound allocations two rings down were kept; the allocator logged `no_eligible_orphan: 12 unbound allocation(s), none steady Active/Paused` on every reconcile for hours. Keeping the best 13 of the 27 was worth ~18% more than the set actually kept. Both paths now rank the whole held set worst-scoring first and shed only the count that cannot be staffed, with the existing halt-risk shield applied uniformly and ties broken toward the unbound allocation so an equal-scoring running worker is never stopped for nothing. Within capacity nothing is shed at all: an orphan that fits gets a worker on the next reconcile. The upstream cause was a second defect. `free_auto` reports workers with an empty filter, which is a statement about the worker, not about the slot: after a restart every worker is unbound while the registry still holds every allocation from before, and the filters are only installed when the allocator next reconciles. A join cycle clearing its readiness gates first saw a fully idle fleet. On mainnet that cycle logged `free_workers=15 total_workers=15` while the prover held 13 Active allocations, proposed a 14-filter join, and 14 seconds later the allocator reported `orphan_count=35`. JOIN_FILTER_COOLDOWN_FRAMES does not cover it: that guards the same filter across overlapping cycles, and this was one cycle proposing fourteen filters it had never proposed before. The join budget is now free workers less the slots already owed to held allocations that nothing has bound yet. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Both capacity paths shed unbound allocations first and score-blind. Which allocation holds a worker is set by bind order, not by value, so the shed set was too — and a leave marks it
Leaving, whichrebind_surplus_by_prioritycan no longer promote, since it takes only steadyActive/Paused. Observed on mainnet: of 27 allocations against 15 workers, the 14 shed included the best-paying ring the node held while lower-scoring bound allocations were kept.The join budget compounded it.
free_autoreports unbound workers, not free slots: after a restart every worker is unbound while the registry still holds every prior allocation, so one cycle proposed 14 joins on top of 13 already held.Rank the whole held set worst-scoring first and shed only the count that cannot be staffed. Halt-risk shield scoped to bound allocations, ties broken toward the unbound. Budget joins as free workers less the slots owed to unbound held allocations.
Stacks on #651 — same block in
provers/lifecycle.rs, merges after.Tests:
cargo test --locked -p quil-engine --libBase:
4eaf1f79