…s, status

Address design-review findings:

- Forward-path heartbeat rule narrowed: the no-expiry exemption now covers only
  the payload-materialization phase (TASK_ACCEPTED -> new TASK_PAYLOAD_READY
  message), bounded by a materialization deadline; heartbeats stay authoritative
  while user code trains, so a wedged process is detected at heartbeat timeout,
  not task timeout. Diagrams updated (mermaid re-validated).
- launch_once=False moves to launch-scoped tokens: the CJ regenerates the token
  in each per-launch bootstrap config and stopping a process invalidates its
  token, so a surviving stale process cannot authenticate against a later launch.
- Plan dependencies now enforce the design's payload-safety prerequisites:
  F3-4 hard-depends on F3-2 + F3-3 (receiver-confirmed, retry-aware outcomes;
  budget-bounded resolution); critical path updated (F3-2/F3-3 parallel behind
  F3-1, one M added, still inside the 10-14 week floor).
- Explicit mapping between lifecycle transfer states and the F3 TransferOutcome
  vocabulary (TRANSFER_COMPLETE <=> COMPLETED; TRANSFER_FAILED <=> FAILED or
  ABORTED; receiver truth applies before the mapping).
- Bookkeeping: PR-0 recorded as landing inside the F3-1 PR (NVIDIA#4853); EX-5 moved
  to Wave 4 (its EP-4 dependency) and assigned P1; CT-5 assigned P1; tier
  accounting line added (36 PRs, CT-4 halves in P1/P2); P0+P1 ~36 engineer-weeks.
- ClientAPIBackendSpec classified as internal (frozen for parallel development,
  not public API in 2.9). Open Questions annotated with their deciding PRs.
- Status set to Approved; Revision 2.2 entry added.
- Add Topic.TASK_PAYLOAD_READY to the frozen protocol vocabulary (follow-up to
  NVIDIA#4856, which merged before the phase-boundary message was introduced).

…ed test helpers

Cleanup pass over the F3 payload-layer commit (4 parallel review angles:
reuse / simplification / efficiency / altitude). No behavior change.

- CONFIRM_EXPECTED is serialized only on terminal replies: since confirmations
  are sent only after terminal serves, the per-chunk advertisement was dead
  weight on the hottest wire message. Terminal branch folded to one if/else.
- Acquisition promoted to the transaction (_acquired_receivers, monotonic set
  with a double-checked add: at most one lock per (transaction, receiver), not
  per chunk). Replaces the per-pass union rebuild across refs in both
  enforce_receiver_budgets and get_acquired_receivers -- acquisition is a
  transaction-level fact, now modeled as one.
- Monitor budget pre-pass pre-filters to transactions that actually have
  budgets (has_receiver_budgets), so the common no-budget case does zero
  per-transaction lock reacquisitions per tick.
- _resolve_receiver_budget now reuses get_positive_float_var and
  check_positive_number instead of hand-rolling both branches.
- Kill-switch comment corrected: per-process (read once; restart to change),
  not "runtime". Redundant pending-confirm pop in enforce_budgets removed
  (_finalize_receiver owns it).
- Test scaffolding deduplicated: the pull/confirm wire builders,
  pull_to_terminal loop, and monitor-suppressed service factory move to
  download_test_utils; the confirm kill-switch fixture moves to a shared
  conftest.py. Removes five helpers copy-pasted across three files.

Deliberate skips: lru_cache for the kill-switch cache (the module global is a
load-bearing test patch point), StreamFuture delegation for TransferWaiter
(different resolve semantics; minimal savings), spec-object for the
transaction kwargs and a general capability-negotiation mechanism (premature
until more consumers/capabilities exist -- noted for later).

The plan file was a point-in-time PR decomposition, not a durable reference:
execution has already deviated from it deliberately (the F3 track shipped as
one consolidated PR; the trainer-engine track was re-scoped), and keeping it
current would mean plan-only doc churn with every such decision. A stale plan
misleads more than no plan; git history preserves it for archaeology. The
design doc -- the durable contract reference -- stays, with its plan
references decoded into work-item names.

…3 payload PR

F3-2/F3-3/F3-4 and "plan:" references decoded against an implementation-plan
document that no longer exists; the feature names say what the ids only
pointed at. Comments now name the mechanisms directly (receiver-confirmed
completion, per-receiver budgets, awaitable transfer facade).

…ore resolving waiters

Fixes a reproduced review batch on the F3 payload layer.

Declared receiver identities are now enforced end to end. Completion
(is_finished, the downloaded_to_all latch), the aggregate outcome, and the
quorum are judged against receiver_ids when declared: a status from an
unexpected receiver can no longer complete -- or certify -- a transfer that a
declared receiver never got. Count-based semantics remain only when
identities are unknown. (Also makes a ref-less transaction never "finished".)

Confirmations are bound to their serve by a per-serve nonce. The terminal
reply carries CONFIRM_NONCE; the confirmation must echo it. The pending-entry
check alone could not distinguish ref lives: a delayed confirmation from a
previous life of a reused ref_id was accepted whenever the new life had its
own pending serve for the same receiver. Confirmations also now honor the
download's secure flag.

Ownership and table registration are one atomic step (_tx_lock nesting
_outcome_lock; no reverse nesting exists). Two concurrent same-id
constructors could previously interleave across the separate critical
sections, leaving the live transaction without outcome ownership while the
retired one recorded its DELETED outcome as the live attempt's verdict.

Outcomes are recorded -- and waiters released -- only after the transaction
settles: terminal progress, done/outcome callbacks, and source release all
complete before wait() can return, so an upper layer that stops the producer
on wait() return can never preempt them.

Idle budgets judge transaction-level activity: a receiver that finished one
ref and went silent was exempt from a sibling ref's acquire budget
(tx-acquired) while having no per-ref idle timestamp there -- escaping both
budgets and pinning the producer to the full TTL.

Superseded transactions (tx_id reuse retirement) release their sources but
suppress transfer-facing emissions (terminal progress, transaction_done_cb,
outcome_cb): their reused tx_id names the live retry, and consumers keyed by
tx_id would misattribute them.

Regression pins for every reproduced scenario: unexpected-receiver
certification, cross-life stale confirmation (with and without a new pending
serve), concurrent same-id creation hammer, settle-before-wait-return,
multi-ref budget escape, superseded-callback silence, secure confirmations.

…ore resolving waiters

Fixes a reproduced review batch on the F3 payload layer.

Declared receiver identities are now enforced end to end. Completion
(is_finished, the downloaded_to_all latch), the aggregate outcome, and the
quorum are judged against receiver_ids when declared: a status from an
unexpected receiver can no longer complete -- or certify -- a transfer that a
declared receiver never got. Count-based semantics remain only when
identities are unknown. (Also makes a ref-less transaction never "finished".)

Confirmations are bound to their serve by a per-serve nonce. The terminal
reply carries CONFIRM_NONCE; the confirmation must echo it. The pending-entry
check alone could not distinguish ref lives: a delayed confirmation from a
previous life of a reused ref_id was accepted whenever the new life had its
own pending serve for the same receiver. Confirmations also now honor the
download's secure flag.

Ownership and table registration are one atomic step (_tx_lock nesting
_outcome_lock; no reverse nesting exists). Two concurrent same-id
constructors could previously interleave across the separate critical
sections, leaving the live transaction without outcome ownership while the
retired one recorded its DELETED outcome as the live attempt's verdict.

Outcomes are recorded -- and waiters released -- only after the transaction
settles: terminal progress, done/outcome callbacks, and source release all
complete before wait() can return, so an upper layer that stops the producer
on wait() return can never preempt them.

Idle budgets judge transaction-level activity: a receiver that finished one
ref and went silent was exempt from a sibling ref's acquire budget
(tx-acquired) while having no per-ref idle timestamp there -- escaping both
budgets and pinning the producer to the full TTL.

Superseded transactions (tx_id reuse retirement) release their sources but
suppress transfer-facing emissions (terminal progress, transaction_done_cb,
outcome_cb): their reused tx_id names the live retry, and consumers keyed by
tx_id would misattribute them.

Regression pins for every reproduced scenario: unexpected-receiver
certification, cross-life stale confirmation (with and without a new pending
serve), concurrent same-id creation hammer, settle-before-wait-return,
multi-ref budget escape, superseded-callback silence, secure confirmations.

Additionally, a property-falsification pass over this batch (six design
guarantees, each attacked with executed counterexample code; 74 attacks run)
found one more instance of the same class: shutdown() cleared _tx_table and
_outcome_owners in separate critical sections, so a new_transaction landing
in the gap had its ownership wiped while staying live in the table -- outcome
unrecordable forever, waiter falsely resolved None. Teardown is now one
atomic step (_tx_lock nesting _outcome_lock, matching new_transaction). The
regression pin discriminates on lock state, not timing, and fails in
milliseconds against the pre-fix code. The other five properties (identity
certification, confirm-serve binding, settle-before-resolve, bounded
silence, fail-closed) held under attack.

Three follow-up review findings on the settlement path:

A raising custom Downloadable.release() could escape transaction_done() --
and with outcome recording moved to the end of settlement, that left the
outcome unrecorded, the waiter pending forever, ownership registered, and
killed the monitor thread. Releases are now individually guarded and
recording runs in a finally block: no exception anywhere in settlement can
prevent the verdict from being recorded and waiters released.

A retry could take ownership of a tx_id while the previous transaction was
mid-settlement (already popped from _tx_table by the monitor, callbacks not
yet run): the retire path never saw it, so its stale transfer-facing
emissions ran against the retry's id. new_transaction now marks a previous
owner superseded when taking ownership, and transaction_done re-reads the
superseded state at each emission gate.

Superseded suppression was also over-broad: transaction_done_cb is a CLEANUP
surface in practice (in-tree callers delete temp files there, e.g.
workspace_cell_transfer._cleanup_transfer_files) and skipping it leaked
files on active tx_id reuse. Cleanup surfaces (transaction_done_cb,
obj.transaction_done, release) now always run; only the transfer-facing
emissions (terminal progress, outcome_cb) are suppressed for superseded
transactions.

Pins: raising release resolves the waiter and consumes ownership off the
monitor thread; mid-settlement retry suppresses outcome_cb/progress while
cleanup still runs; retirement fires the cleanup callback but not outcome_cb.

Follow-up review found the supersede mechanism unsound in both directions:
the boolean gate is a check-then-emit race (a retry can take ownership after
the old generation passes the check but before the callback fires -- there is
no safe interleaving without holding a lock across user callbacks), and
always-running transaction_done_cb misattributes: it is a completion surface
(FOBS DOWNLOAD_COMPLETE_CB), not just cleanup, so an old generation reporting
DELETED under an id a live retry owns is wrong.

Both dissolve under strict generation serialization: new_transaction may not
take ownership of a reused tx_id until every previous generation has fully
settled. A live prior transaction is retired and settled inline; a
mid-settlement one (popped from _tx_table, callbacks in flight) is awaited on
its settled event, set in transaction_done's finally after outcome recording.
Every terminal emission of an old generation therefore happens strictly
before its retry exists -- nothing needs to be suppressed, and the
superseded flag, its gates, and the retire special case are deleted.
Ownership entries in _outcome_owners now double as the mid-settlement
markers; recording consumes them.

Guards: a settlement callback that synchronously reuses its own tx_id gets
an immediate error instead of a self-deadlock; a retry waiting on a hung
settlement fails loudly after a bounded wait (60s).

Property falsification against the new design found and fixed two gaps
before review: a settled-but-unconsumed owner (outcome recording itself
failed) was reclaimed-never -- Event.wait on a set event returns True
instantly, so the bounded-wait escape was dead code and a retry hot-spun on
the global locks forever; owners are now reclaimed when already settled.
And shutdown() cleared ownership markers before its deferred settlements
ran, so a same-id retry in that window registered while old-generation
emissions were still to come; shutdown now keeps the markers (settlement
consumes them) and drops verdicts via a per-transaction record-forbidden
flag instead -- nothing records after shutdown, waiters still resolve.

Also: deleted dead _Transaction.timed_out() (settled without unlinking from
_tx_table -- the one path that could double-settle); outcome computation
moved inside try so a failure there cannot poison the tx_id; the settlement
wait deadline re-arms per generation; waiter semantics documented and
pinned (waiters bind to the generation that records while they wait).

Scope note, documented in code and description: serialization covers the
terminal surfaces. A data-plane event already executing at retirement (an
in-flight serve's produce(), a monitor budget pass) may finish after the
retry registers; stragglers cannot alter any recorded verdict.

shutdown() resolves the waiters it sees and keeps ownership markers so
same-id retries serialize behind the deferred settlements -- but that
surviving marker makes the tx_id look live to get_transfer_waiter: a waiter
created in the window between shutdown's locked teardown and a deferred
settlement consuming ownership was parked, and the record-forbidden branch
of _record_outcome then consumed ownership without draining it. The waiter
hung forever, violating "waiters never hang".

The forbidden branch now abandons any window waiters with None, the same
verdict shutdown gave the waiters it saw. Every path that consumes
ownership now drains waiters: record resolves with the outcome, reclaim and
the forbidden branch abandon with None, shutdown resolves wholesale.

Pin holds the deferred settlement open with a gated callback, parks a
waiter mid-window, and asserts it resolves to None; verified to fail
(bounded) with the drain removed.

…reuse

Five review rounds attacked the reused-tx_id retry design; four falsified
its generation boundary (mid-settlement supersede, flag-gated suppression
races, unbounded retirement, in-flight budget-pass emissions, shutdown
window waiters). Each fix was correct, but the pattern was defending a
boundary that naming can delete: when two attempts never share an id, a
late emission from a dying attempt names that attempt truthfully forever,
and misattribution becomes unrepresentable instead of prevented.

Grounding facts: the tx_id never crosses the wire in the pull protocol
(receivers pull by ref_id), so retry idempotency always belonged to the
application-level transfer id carried in caller metadata -- in any design.
And id reuse never existed as behavior: the tx_id parameter has been
dormant since introduction (every caller takes a fresh uuid). The reuse
contract was a plan for the not-yet-written trainer engine, with zero
consumers to migrate.

new_transaction now registers in one atomic step and raises ValueError on
a duplicate id while it is live, settling, or its receipt is retained
(TX_OUTCOME_TTL bounds the exclusion window for caller-supplied ids).
Deleted with the reuse contract: generation serialization (settled events,
mid-settlement waits, settler threads, SETTLEMENT_WAIT_TIMEOUT, the
settlement-callback reentrancy guard), dead-owner reclaim and its waiter
abandonment, receipt-wipe-on-registration, and the record_forbidden
marker dance in shutdown (which reverts to the simple atomic clear; the
owner guard alone gates post-shutdown recording, and a waiter arriving in
the deferred-settlement window now resolves None immediately instead of
parking). Waiters trivially bind to the single attempt their id names.

Kept, rationale unchanged: receipts and TTL, TransferWaiter, per-receiver
budgets, receiver confirms including the per-serve nonce (ref_id reuse via
PASS_THROUGH re-emission is independent of retries), tombstones, the
settle-then-resolve ordering, exception-guarded callbacks, and the
activity gate -- demoted from misattribution defense to verdict
completeness: an operation finishing during termination is counted in the
outcome, and nothing emits against a settled transaction.

The duplicate check expires receipts inline, so the exclusion window for a
caller-supplied id is exactly TX_OUTCOME_TTL (not TTL plus a sweep cycle).
The execution-modes design doc is aligned: tx_id never equals transfer_id;
each attempt gets a fresh download_tx_id and the manifest carries the
transfer_id mapping.

Tests updated to pin the new contract: duplicate ids rejected across the
whole lifetime (live, settling, receipted) and freed by receipt expiry;
racing same-id constructors resolve to exactly one winner and clean
errors; settlement callbacks may create new transactions but not their
own id; a recording failure leaves the id excluded and shutdown releases
its waiter; shutdown-window waiters resolve immediately; the in-flight
budget-pass drain now pins that enforcement verdicts are counted in the
receipt. Net -293 lines.

Review found the composition gap in the attempt-scoped duplicate check: a
settlement drain that times out (OP_DRAIN_TIMEOUT) deliberately proceeds
with an operation still in flight, and a caller-supplied tx_id becomes
registrable again once its receipt expires (TX_OUTCOME_TTL) -- so a leaked
operation that resumes after both windows could emit progress/failure
under an id that now names a NEW transaction.

Every terminator now registers a drain-timeout leak (tx with in-flight
operations after settlement) in _leaked_ops; new_transaction rejects the
id -- independently of receipt state -- until the leaked operation count
reaches zero, and the monitor reaps entries whose operations exited. The
id becomes reusable only after receipt expiry AND all old operations
exited; the default fresh uuid never collides with anything.

Description wording tightened to match the implementation: ids are
single-use while known (live, settling, receipted, or leaked), not
"never reused, forever".

Pin: an operation held past a shortened drain timeout keeps the id
rejected even with the receipt force-expired, and the id frees the moment
the operation exits.

Comment/docstring share of download_service.py had grown to a third of the
file across the review rounds -- much of it review-history narration and
correctness arguments addressed to reviewers, which are noise once merged
(that knowledge lives in commit messages and the regression pins).

Rules applied: comments state constraints the code cannot show (lock
ordering, unlink-before-settle, why callbacks run outside locks, wire
semantics); each invariant is documented at exactly one home (table
invariants on the class attributes, contracts in method docstrings) with
other sites shortened to a pointer; review archaeology and
justification-of-change prose deleted. Public API docstrings and test
docstrings (each pin documents the bug it guards) are untouched.

Review found the leak exclusion's shutdown gap: shutdown() clears the
ownership and receipt exclusions up front, but leak registration ran only
AFTER the deferred settlement -- so a drain-leaked operation's tx_id
looked free for that whole window (unbounded if a settlement callback
hangs), and a replacement could register while the leaked operation was
still able to emit under the id. delete_transaction and the monitor were
not affected: their ownership marker covers settlement and their receipt
covers the tail.

shutdown() now pre-registers every registered owner (live and
mid-settlement) as a potential leak in the same atomic teardown section,
before clearing ownership. The per-terminator hook becomes a sync
(_update_leaked_ops): it keeps the exclusion while operations are in
flight and releases it once none are, so pre-registered ids free normally
after a clean drain. Pin drives the exact reviewer sequence: leaked op +
shutdown mid-settlement -> same-id registration rejected, still rejected
after shutdown returns, freed the moment the operation exits.

Also from review: explicit pin that a CONFIRM-keyed message dispatches to
confirm handling and never falls through to pull handling (a fall-through
would re-serve from scratch); duplicate receiver_ids now log a warning
(almost certainly a caller bug) with the dedup pin extended to assert it;
make_confirm_test_service renamed to make_service_no_monitor (it was
never confirm-specific); conftest notes that its autouse confirm-switch
fixture is directory-wide.

Review round 8 falsified the shutdown pre-registration's release condition:
a marker was releasable whenever _active_ops == 0, but settlement callbacks
are not operations -- so with the old transaction blocked inside
transaction_done_cb (zero ops), a same-id constructor could pop the marker
mid-settlement, register, and then the old outcome_cb fired under an id
naming the replacement.

The marker is now two-state, as the reviewer prescribed: releasable only
when the transaction's settlement has completed (a flag set in
transaction_done's finally, after outcome recording -- nothing emits after
it) AND no operations are in flight. All three release paths enforce the
conjunction: the duplicate check in new_transaction, the per-terminator
_update_leaked_ops sync, and the monitor reap. A registration that slips
between the flag and the terminator's sync is legal (settlement is fully
done), and the sync's identity check keeps it from ever touching the
replacement's state.

Pin drives the reviewer's exact clean-settlement sequence: shutdown with a
gated transaction_done_cb and zero operations -> same-id registration
rejected mid-settlement, accepted once settlement completes.

Review round 9 composed two timing facts into an exposure: the outcome
timestamp is captured before the settlement callbacks run, so a settlement
slower than TX_OUTCOME_TTL recorded a receipt that was expired at birth --
and the inline expiry check then handed the id to a same-id constructor in
the gap after ownership was consumed but before the terminator's marker
sync ran. The old transaction's marker then landed on top of the live
replacement while its leaked operation could still emit.

Two changes, per the reviewer's recommendation:

The termination marker is now installed by _delete_tx itself, atomically
with the unlink, for EVERY terminator -- not just shutdown. One mechanism
covers the entire termination window (settlement plus any drain-leaked
tail) on every path; the ownership and receipt exclusions still exist but
no longer carry any window alone. Shutdown's separate owner
pre-registration loop became redundant and is removed. The overlap state
can no longer form: the marker exists before ownership is consumed, and
releases only on settlement-complete AND zero operations.

Receipts are re-stamped at recording time: the TTL retention window starts
when the receipt becomes queryable, matching the documented "kept 30 min"
contract, so a slow settlement can never record an already-expired receipt.

Pins: slow-settlement receipt is fresh from recording and excludes the id;
the reviewer's aged-receipt-plus-leaked-op sequence stays excluded end to
end with no overlap; the marker exists mid-settlement on the delete path
(not just shutdown).

…erdict

Review round 10 falsified the previous fallback twice over: a one-shot
outcome-computation failure skipped every cleanup emission and the source
release (waiter resolved, 70GB pinned), and a persistent failure escaped
the finally because the fallback REUSED the failing function -- stranding
the waiter, leaving ownership and the termination marker unreleasable, and
able to kill the monitor thread. The regression pin masked both by raising
only on the first call and never asserting release.

The ceremony is now three independent phases:

Verdict: computed under its own guard; a failure is contained and no
longer skips anything downstream. Emissions that need the verdict
(outcome_cb) are gated on having one; cleanup emissions
(obj.transaction_done, transaction_done_cb, terminal progress) run
regardless.

Source release: its own finally phase -- no failure in the verdict or the
callbacks can pin the sources in memory.

Recording: runs last and cannot raise. The fallback is direct fail-closed
dataclass construction (never compute_transfer_outcome), each step is
guarded, and the settlement-complete flag is set unconditionally, so
ownership is consumed, waiters resolve, and the termination marker stays
releasable no matter what happened above.

Codex review refinements on the same patch: the fail-closed verdict is
built in the computation exception handler itself (a _fail_closed_outcome
helper) with full transaction metadata and a dedicated
COMPUTATION_FAILED reason, and flows through outcome_cb like any verdict
-- the callback contract holds on this path; the recording-phase fallback
remains as a last-resort belt via the same helper.

Codex round 2 returned clean and contributed test-only additions, kept:
the fail-closed outcome preserves min_receivers/receiver_ids for both
outcome_cb and the waiter, and outcome_cb runs before source release.

Pin hardened to the reviewer's spec: compute_transfer_outcome raising
PERSISTENTLY -> waiter resolves fail-closed, ownership consumed, sources
released, per-object cleanup fired, no exception escapes the terminator,
and a new transaction registers cleanly afterward.

The dict began (review round 7) as a registry of drain-LEAKED OPERATIONS;
rounds 8-9 grew it into the general termination marker installed at unlink
for every terminator -- most entries never involve a leak, and the values
are transactions, not operations. The name now says what the duplicate
check reads it as, matching its error message: the id's previous
transaction "has not fully terminated".

_update_leaked_ops -> _sync_termination_marker (release or sustain after
settlement), _reap_leaked_ops -> _reap_termination_markers. Mechanical
rename; no behavior change. The isolated-service test factory override is
renamed with it, keeping test isolation intact.