diff --git a/docs/design/client_api_execution_modes.md b/docs/design/client_api_execution_modes.md index 71f3380b84..d1914338d4 100644 --- a/docs/design/client_api_execution_modes.md +++ b/docs/design/client_api_execution_modes.md @@ -2,12 +2,14 @@ ## Status -Proposal / design note. Epic: **FLARE-2698** (Client API and 3rd party integration refactoring, fix version 2.9.0). +**Status: Approved** — implementation in progress for 2.9 (see the companion implementation plan, `client_api_execution_modes_plan.md`). Epic: **FLARE-2698** (Client API and 3rd party integration refactoring, fix version 2.9.0). Revision 2 (2026-07-01): incorporates review feedback — universal session setup, owner-death and receive-side contracts, forward-path payload lifecycle, cleanup policy definition, receiver-confirmed terminal outcome, per-receiver transfer budgets, configuration surface, attach auth hardening, and a re-sequenced migration plan. Revision 2.1 (2026-07-06): adds the filters-imply-materializing-hop rule to the payload lifecycle (per-(task,direction) granularity), conversion-filter placement details (site-scope filter ordering, DXO-meta round-trip state, mixed-format cost and bf16 limit, and what selects the regime — `server_expected_format` surviving as a recipe knob), the TensorStream disposition, and a CCWF caveat cross-referencing the materializing-hop rule (a wired conversion filter moves the producer role from the subprocess to the CJ). Claims verified against code. +Revision 2.2 (2026-07-07): status set to Approved. Forward-path heartbeat exemption narrowed to the payload-materialization phase via a new TASK_PAYLOAD_READY control message (heartbeats stay authoritative while user code trains); `launch_once=False` moves to launch-scoped tokens (fresh binding material per launch, invalidated at process stop); explicit mapping between the lifecycle transfer states and the F3 `TransferOutcome` vocabulary; `ClientAPIBackendSpec` classified as internal; Open Questions annotated with their deciding PRs. + ## Background NVFlare supports the Client API so training scripts can interact with FLARE through a small surface: @@ -226,7 +228,9 @@ sequenceDiagram C->>E: task E->>T: TASK_READY (task_id, FLModel ref) T->>E: TASK_ACCEPTED - Note over T: pull task payload, then train() + Note over T: pull task payload + T->>E: TASK_PAYLOAD_READY + Note over T: train() T-->>E: LOG / HEARTBEAT T->>E: RESULT_READY (result_id, transfer_id, manifest) E->>T: RESULT_ACCEPTED (control ack, not payload done) @@ -239,7 +243,10 @@ sequenceDiagram ``` Sequence — `launch_once=False`. A fresh process per task; the HELLO handshake repeats -each round (the job-scoped launch token is reused). The key difference: the per-task +each round with fresh binding material — the CJ regenerates the launch token in the +bootstrap config it already writes per launch, and stopping the previous process +invalidates its token, so a stale process that survived termination cannot +authenticate against a later launch (the token is launch-scoped, not job-scoped). The key difference: the per-task process stop **waits for the terminal transfer outcome** before killing the process — this is what replaces the legacy `download_complete_timeout` deferred stop: @@ -250,8 +257,8 @@ sequenceDiagram participant T as Trainer loop each task C->>E: task - E->>T: write config + Popen(...) [new process] - Note over T: flare.init() (job-scoped token reused) + E->>T: write config (fresh launch token) + Popen(...) [new process] + Note over T: flare.init() (per-launch token) T->>E: HELLO (launch token) E->>T: HELLO_ACCEPTED (fresh session) E->>T: TASK_READY (task_id, FLModel ref) @@ -260,7 +267,7 @@ sequenceDiagram Note over E,T: WAIT_TRANSFER_COMPLETE (hold process until terminal) E->>C: result E->>T: SHUTDOWN - Note over E,T: stop THIS process group,
next task launches a new process + Note over E,T: stop THIS process group and invalidate its token,
next task launches a new process end ``` @@ -307,6 +314,7 @@ sequenceDiagram C->>E: task E->>T: TASK_READY (task_id, FLModel ref) T->>E: TASK_ACCEPTED + T->>E: TASK_PAYLOAD_READY (payload pulled) T-->>E: LOG / HEARTBEAT (session lease) Note over T: train() T->>E: RESULT_READY (result_id, transfer_id, manifest) @@ -410,7 +418,9 @@ subprocess behavior.) ``` CJ -> trainer : TASK_READY (task id, task name, FLModel ref, params) trainer -> CJ : TASK_ACCEPTED -...trainer pulls task payload, trains... +...trainer pulls task payload... +trainer -> CJ : TASK_PAYLOAD_READY (task id — payload materialized or lazily held, training begins) +...trainer trains (heartbeats continue autonomously)... trainer -> CJ : TASK_FAILED (task id, reason — e.g. task payload download failed) [failure path] trainer -> CJ : RESULT_READY (result_id, transfer_id, manifest) CJ -> trainer : RESULT_ACCEPTED (or RESULT_REJECTED) @@ -460,7 +470,7 @@ TASK_ACCEPTED acknowledges control receipt of the task message only — it does - In external_process the executor has two liveness signals: process exit (waitpid on the process tree it owns) and heartbeat. Process exit is authoritative for "trainer is gone"; heartbeat covers a live-but-wedged trainer. - In attach the heartbeat lease is the only liveness signal the executor has. - **Precedence rule for transfers (result path):** active data-plane transfer progress counts as session liveness. While a session has an in-flight result transfer (WAIT_PAYLOAD_ACQUIRED/WAIT_TRANSFER_COMPLETE), the session lease does not expire on missed control heartbeats alone; the transfer's own idle/progress policy (shared F3 layer) governs, and only transfer failure/timeout or an explicit abort ends the session. This resolves the otherwise-contradictory pair "heartbeat timeout invalidates the session" vs "the executor must not revoke the session before terminal payload state" — and heartbeat false-positives during multi-GB result uploads are one failure class this design removes. (The legacy stack solved this with a dedicated STREAM_PROGRESS topic feeding transfer waits, deliberately separate from peer liveness; the same separation is preserved here, just owned by the shared transfer layer.) -- **Precedence rule for the forward (task-delivery) path:** the same false-positive class exists in the *other* direction and needs its own rule, because the CJ cannot see forward bytes. When TASK_READY carries a lazy ref to a multi-GB global model, the control rank pulls those bytes **directly from the producer** (server job process); the CJ that owns the session/heartbeat lease is only a relay and observes no forward transfer (the producer's progress callback fires at the server, not the CJ). A control rank busy materializing 5 GB inside `flare.receive()` therefore has no in-flight-transfer signal to feed the rule above. Rule: **the CJ must not revoke the session on control-heartbeat timeout while a TASK_ACCEPTED is outstanding** — an accepted-but-not-yet-completed task means the trainer is legitimately working (materializing the payload and/or training), and forward materialization is bounded by the trainer-local pull policy (the shared F3 idle/timeout on the trainer→producer transfer), not by the control heartbeat. This is the forward analog of the result-path rule and closes the same 5GB-materialization false-positive that `progress_aware_streaming.md` documents on `task_payload_download`. +- **Precedence rule for the forward (task-delivery) path:** the same false-positive class exists in the *other* direction and needs its own rule, because the CJ cannot see forward bytes. When TASK_READY carries a lazy ref to a multi-GB global model, the control rank pulls those bytes **directly from the producer** (server job process); the CJ that owns the session/heartbeat lease is only a relay and observes no forward transfer (the producer's progress callback fires at the server, not the CJ). A control rank busy materializing 5 GB inside `flare.receive()` therefore has no in-flight-transfer signal to feed the rule above. Rule: **the CJ must not revoke the session on control-heartbeat timeout during the payload-materialization phase — from TASK_ACCEPTED until the trainer sends TASK_PAYLOAD_READY (or TASK_FAILED)** — because materialization is bounded by the trainer-local pull policy (the shared F3 idle/timeout on the trainer→producer transfer), not by the control heartbeat; the CJ additionally bounds the phase with a materialization deadline derived as an envelope over that pull policy, so a trainer that dies mid-pull without emitting TASK_FAILED still fails the task instead of holding the session open. The exemption ends at TASK_PAYLOAD_READY: **while user code trains, heartbeats stay authoritative** — the trainer engine's heartbeat thread runs autonomously alongside training (the same autonomy the send path guarantees above), so a healthy trainer keeps its lease through arbitrarily long training and a wedged or dead one is detected at the normal heartbeat timeout rather than at the task timeout. This matters most in attach, where the heartbeat lease is the only liveness signal. This is the forward analog of the result-path rule and closes the same 5GB-materialization false-positive that `progress_aware_streaming.md` documents on `task_payload_download` — without disabling liveness for the (much longer) training phase. #### CJ Failure (Owner Death) @@ -512,7 +522,7 @@ DONE_CLEANUP_ALLOWED [terminal — producer may exit and free payload] ABORT at any state ────────────────────────────────► ABORTED [terminal] ``` -**Terminal states are DONE_CLEANUP_ALLOWED (success), TRANSFER_FAILED, RESULT_REJECTED, and ABORTED.** (TRANSFER_COMPLETE / TRANSFER_FAILED are the transfer-layer outcomes the machine consumes; TRANSFER_COMPLETE moves the machine to DONE_CLEANUP_ALLOWED. Earlier drafts used the two vocabularies interchangeably; they are now distinct.) RESULT_ACCEPTED is reached well before cleanup is allowed — that gap is the whole point: an accepted control message is not a completed payload transfer. +**Terminal states are DONE_CLEANUP_ALLOWED (success), TRANSFER_FAILED, RESULT_REJECTED, and ABORTED.** (TRANSFER_COMPLETE / TRANSFER_FAILED are the transfer-layer outcomes the machine consumes; TRANSFER_COMPLETE moves the machine to DONE_CLEANUP_ALLOWED. Earlier drafts used the two vocabularies interchangeably; they are now distinct.) The transfer layer itself reports a third, F3-native vocabulary: the aggregate `TransferOutcome.status` values COMPLETED / FAILED / ABORTED (`transfer_outcome.py`, reusing the TransferProgressState terminal set). The mapping at that boundary is fixed and total: TRANSFER_COMPLETE ⇔ COMPLETED; TRANSFER_FAILED ⇔ FAILED **or** ABORTED (the raw `done_status` and `reason` ride along for diagnostics, but the lifecycle machine does not branch on them). Receiver truth is applied *before* the mapping: a transaction deleted by routine cleanup after every expected receiver succeeded resolves COMPLETED, not ABORTED. RESULT_ACCEPTED is reached well before cleanup is allowed — that gap is the whole point: an accepted control message is not a completed payload transfer. Logical ownership: @@ -576,6 +586,7 @@ The forward direction — global model from server/CJ to the trainer — is wher - **TASK_READY is control-only.** It carries the task metadata and FLModel lazy refs; TASK_ACCEPTED acknowledges the control message, before any payload materialization — the same "accepted ≠ transferred" caveat as RESULT_ACCEPTED. - **The trainer's pull is governed by the same shared transfer policy.** `flare.receive()` materializes (or lazily holds) the model by pulling through the shared F3 path; idle/progress/timeout policy is the shared layer's, and the upstream producer (CJ relay or server job process) is held to the same producer-liveness rule — its refs stay alive until the trainer's pull reaches a terminal state. - **Trainer-side download failure is explicit.** If the task payload pull fails or times out, the trainer backend sends TASK_FAILED with the task id and reason; the executor fails or retries the task at the workflow's discretion. There is no silent hang and no ambiguous half-delivered task. +- **Materialization completion is explicit.** When `flare.receive()` hands the task to user code (payload materialized, or lazily held), the trainer backend sends TASK_PAYLOAD_READY with the task id. TASK_PAYLOAD_READY or TASK_FAILED ends the payload-materialization phase — and with it the forward-path heartbeat exemption (see Heartbeat and Liveness); from that point the session lease is governed by heartbeats as normal while user code trains. - **No blind resend.** The CJ does not resend TASK_READY on a timer while the session is alive (Cell request/reply plus TASK_READY idempotency replace the Pipe resend loop that caused duplicate-delivery races). Relationship to `progress_aware_streaming.md`: that design's Phase-1 wait policies are hosted today in TaskExchanger (CJ-side forward waits) and FlareAgent (trainer-side reverse waits) over an internal Pipe topic — exactly the components this design retires. The progress-tracking substrate it defines (per-transfer progress in the shared F3 layer) is the part that survives and is what the waits here consume; the Pipe-topic delivery and TaskExchanger/FlareAgent wait owners die with the legacy stack. The forward-path contract above is the re-homed replacement: readiness comes from HELLO, delivery from Cell request/reply, and materialization waits from the shared transfer layer instead of resend suppression. @@ -733,6 +744,8 @@ silently dropped. `task_script_path`/`task_script_args` (the in_process trainer entry point), the task-name mapping, and the memory knobs are carried forward from today's InProcessClientAPIExecutor/ScriptRunner so existing jobs map without loss. +**`ClientAPIBackendSpec` is internal.** The mode backends behind the executor implement a single `ClientAPIBackendSpec` interface (in `client_api_executor.py`). Its surface is frozen early so the mode backends can be built in parallel, but it is an internal extension interface, not public API in 2.9: users select and configure modes only through the ClientAPIExecutor/ScriptRunner arguments above, and no third-party backend registration point is exposed (see the rejected general Client API Launcher under Alternatives Considered). + **No parameter converters on the executor (per FLARE-2698).** The `params_exchange_format`, `params_transfer_type`, `server_expected_format`, and `from/to_nvflare_converter_id` arguments of the legacy executors are intentionally absent. Conversion between the framework-agnostic @@ -811,7 +824,7 @@ One executor component, no pipes, no launcher, no MetricRelay: LOG messages arri ### Observability -Each session exposes its lifecycle state for operators: session state (waiting HELLO / idle / task active / waiting transfer), current task_id/result_id/transfer_id, per-receiver transfer progress (from the shared layer's progress events), and last heartbeat. These surface through the standard job stats/log channels (CJ logs at state transitions with the ids above; stats pollable via the existing cell/job info mechanisms), so "why is this producer still alive" and "which receiver is stalled" are answerable from the site without a debugger. +Each session exposes its lifecycle state for operators: session state (waiting HELLO / idle / task materializing / task active / waiting transfer), current task_id/result_id/transfer_id, per-receiver transfer progress (from the shared layer's progress events), and last heartbeat. These surface through the standard job stats/log channels (CJ logs at state transitions with the ids above; stats pollable via the existing cell/job info mechanisms), so "why is this producer still alive" and "which receiver is stalled" are answerable from the site without a debugger. ## Scenario Coverage @@ -1020,12 +1033,13 @@ Core external_process and rank contract (steps 3–4): Session and failure contracts (steps 3–4): -- external_process: a HELLO carrying a wrong/stale launch token is rejected (HELLO_REJECTED) and a stale trainer from a previous run cannot bind; attach: a bad token proof fails at HELLO_PROOF (see the attach auth tests) +- external_process: a HELLO carrying a wrong/stale launch token is rejected (HELLO_REJECTED) and a stale trainer from a previous run cannot bind (under `launch_once=False`, the previous launch's token is invalidated at process stop, so a surviving per-task process cannot bind to the next launch); attach: a bad token proof fails at HELLO_PROOF (see the attach auth tests) - HELLO with a mismatched protocol version fails fast with a clear error - duplicate TASK_READY delivery does not double-deliver the task to user code - trainer-side task payload download failure produces TASK_FAILED, not a hang - CJ kill (SIGKILL) leads to trainer process-group self-termination within the grace period; CP reaping covers a disabled trainer-side grace - heartbeat loss during an active multi-GB transfer does not revoke the session while the transfer is progressing; transfer failure/timeout does +- heartbeat loss during payload materialization (TASK_ACCEPTED outstanding, TASK_PAYLOAD_READY not yet sent) does not revoke the session; sustained heartbeat loss after TASK_PAYLOAD_READY, while user code trains, does - `flare.receive()` returns None on SHUTDOWN/job end; raises the session exception on ABORT and on session loss Payload lifecycle (steps 2, 4): @@ -1050,11 +1064,13 @@ CCWF (step 6): ### Open Questions -- Exact public argument names and migration aliases. -- Scheduler batch helper library vs. standard wrapper component. -- Exact factoring of shared code with IPCAgent/IPCExchanger. -- Compatibility flag for selecting the legacy Pipe path during migration. -- Whether partial fan-out (N-1 of N receivers succeeded) should be surfaceable to CCWF workflows as a usable result or always a task failure. +Each open question is tracked to the PR that decides it (plan PR ids); none blocks the interface freezes already landed. + +- Exact public argument names and migration aliases — decided at the EX-2 interface freeze (the frozen `ClientAPIExecutor` constructor is the decision record) and the EX-3 ScriptRunner wiring. +- Scheduler batch helper library vs. standard wrapper component — deferred to Migration Plan step 7 (Future Enhancements); not 2.9-blocking. +- Exact factoring of shared code with IPCAgent/IPCExchanger — decided during the trainer-engine PR (TE-4), where the shared session/receive machinery gets its final home. +- Compatibility flag for selecting the legacy Pipe path during migration — decided in EX-3/EP-4: whether legacy selection stays purely class-name-based (per the Compatibility section's keep-legacy story) or also gets an explicit flag. +- Whether partial fan-out (N-1 of N receivers succeeded) is surfaceable to CCWF workflows as a usable result or always a task failure — decided by F3-3's quorum surface (the plan requires F3-3 to settle it) and surfaced to workflows in CC-1. ## Migration Plan diff --git a/docs/design/client_api_execution_modes_plan.md b/docs/design/client_api_execution_modes_plan.md index 636d574360..68c5631f49 100644 --- a/docs/design/client_api_execution_modes_plan.md +++ b/docs/design/client_api_execution_modes_plan.md @@ -1,6 +1,6 @@ # Client API Execution Modes — 2.9 Implementation Plan -Companion to `client_api_execution_modes.md` (design). Tracks Epic **FLARE-2698** (Client API and 3rd party integration refactoring, 2.9.0). Decomposes the design's 8-step Migration Plan into **36 PRs** with dependencies, sizes, and a release cut line. Scoped against the codebase post-2.8.0; granularity calibrated against the repo's merged-PR history (2026-07-01, see Calibration below). +Companion to `client_api_execution_modes.md` (design). Tracks Epic **FLARE-2698** (Client API and 3rd party integration refactoring, 2.9.0). Decomposes the design's 8-step Migration Plan into **36 PRs** with dependencies, sizes, and a release cut line (the wave plan lists 37 entries; PR-0 is not a separate PR — the doc set lands inside the F3-1 PR, #4853). Scoped against the codebase post-2.8.0; granularity calibrated against the repo's merged-PR history (2026-07-01, see Calibration below). Size guide: **S** <300 changed LOC, **M** 300–800, **L** 800–2000. @@ -24,8 +24,8 @@ Tracks: **F3** payload layer · **EX** executor/ScriptRunner · **TE** trainer e | PR | Track | Size | Notes | |---|---|---|---| -| PR-0 Land design doc + this implementation plan in docs/design/ | all | S | **Lands first, merges fast** — the design is already approved; this just puts the reference material in-repo so every subsequent PR links it. Interface-freeze sign-offs happen on TE-1/EX-2/F3-1 themselves | -| F3-1 Aggregate all-receivers terminal transfer outcome | F3 | M | Interface freeze #3. Purely additive next to FINISHED/TIMEOUT/DELETED | +| PR-0 Land design doc + this implementation plan in docs/design/ | all | S | **Lands with F3-1 in a single PR (#4853)**, not as a separate docs-only PR — the reference material merges together with the first frozen contract, and every subsequent PR links it. Interface-freeze sign-offs happen on TE-1/EX-2/F3-1 themselves | +| F3-1 Aggregate all-receivers terminal transfer outcome | F3 | M | Interface freeze #3. Purely additive next to FINISHED/TIMEOUT/DELETED. Ships as #4853, carrying PR-0's docs | | TE-1 Protocol vocabulary (`client/cell/defs.py`) | TE | S | Interface freeze #1 — the Cell control-protocol Topics/MsgKeys/CHANNEL/version only. The auth **mechanism** is deliberately NOT frozen here: the token is 3 roles (rendezvous, anti-mixup, auth) and whether external_process needs a *secret* HMAC proof at all is a host-trust decision (single-tenant: no; multi-tenant: yes) that EP-3 makes. So the proof helpers (TokenScope, compute/verify_hello_proof, combine_nonces) land with **EP-3**, and the generic generators (generate_session_token/nonce, token_digest) go to the fuel/sec consolidation (**FLARE-3017**). The stateful SessionTokenManager is attach-only (**AT-2**) | | EX-2 ClientAPIExecutor skeleton + backend spec + analytics-event ownership | EX | M | Interface freeze #2 | | TE-2 Bootstrap config schema + NVFLARE_CLIENT_API_CONFIG resolution | TE | M | Additive ConfigKeys; consumes EP-1's 0600 writer. Kept separate from TE-1: touches legacy-shared `client/config.py`, different revert profile | @@ -38,8 +38,8 @@ Tracks: **F3** payload layer · **EX** executor/ScriptRunner · **TE** trainer e | PR | Track | Size | Depends | |---|---|---|---| | F3-2 Receiver-confirmed completion + retry-aware accounting | F3 | M | F3-1. The version-skew wire change — lands as early as possible for maximum soak; capability-flag gated, both skews interop-tested | -| F3-3 Per-(transfer, receiver) acquire/idle budgets | F3 | M | F3-1. Unconditional per-receiver activity tracking. Must also settle the quorum surface for fan-out: workflows with min_responses-style policy (k-of-N receivers suffices) need either an optional min_receivers on the transaction/facade or a documented pattern of evaluating TransferOutcome.refs against their own threshold — `completed` stays the strict all-receivers certificate either way | -| F3-4 Awaitable producer transfer facade + PAYLOAD_ACQUIRED + post-completion linger | F3 | M | F3-1 (and F3-2 for the linger's receiver-confirmed path). Must CHAIN existing DOWNLOAD_COMPLETE_CB, not replace. Folds in the former F3-5 bounded post-completion linger (a leaf with no independent consumer — only this facade's release path uses it); F3-2 still lands as its own isolated PR so the wire-skew change is not welded here | +| F3-3 Per-(transfer, receiver) acquire/idle budgets | F3 | M | F3-1; blocks F3-4, whose aggregate outcome resolves in bounded time only through these budgets (a stalled receiver is marked failed without waiting for the full transaction TTL). Unconditional per-receiver activity tracking. Must also settle the quorum surface for fan-out: workflows with min_responses-style policy (k-of-N receivers suffices) need either an optional min_receivers on the transaction/facade or a documented pattern of evaluating TransferOutcome.refs against their own threshold — `completed` stays the strict all-receivers certificate either way | +| F3-4 Awaitable producer transfer facade + PAYLOAD_ACQUIRED + post-completion linger | F3 | M | F3-1, F3-2, F3-3 — all hard. The facade's returns-means-delivered guarantee holds only over receiver-confirmed, retry-aware outcomes (F3-2), and its aggregate outcome resolves in bounded time only with the per-receiver budgets (F3-3); a facade over producer-served EOF would reintroduce the silent-truncation gap. Must CHAIN existing DOWNLOAD_COMPLETE_CB, not replace. Folds in the former F3-5 bounded post-completion linger (a leaf with no independent consumer — only this facade's release path uses it); F3-2 still lands as its own isolated PR so the wire-skew change is not welded here | | EX-3 in_process backend (consolidate InProcessClientAPIExecutor) | EX | L | EX-2. Behavior-parity bar: "nothing user-visible" | | TE-3 TrainerCellSession engine (handshake, heartbeat, owner-death, trainer-side authenticated teardown) | TE | L | TE-1, TE-2. Injectable clock + kill hook; AT owner co-reviews the teardown-auth tests | | EP-5 CP-side orphan reaping of trainer PGIDs | EP | M | EP-2. PID-reuse guard via start-time record | @@ -48,10 +48,9 @@ Tracks: **F3** payload layer · **EX** executor/ScriptRunner · **TE** trainer e | PR | Track | Size | Depends | |---|---|---|---| -| TE-4 TrainerCellSession task/result contracts (receive queue + TASK_READY idempotency + TASK_FAILED; RESULT_READY flow + terminal-outcome blocking + fan-out drain/shutdown gating) | TE | L | TE-3; F3-4 via a narrow stubbed wait-protocol interface. Merged from two Ms: same class, same owner, no independent consumer of either half. Guard: split fan-out drain back out if the diff passes ~1800 LOC | +| TE-4 TrainerCellSession task/result contracts (receive queue + TASK_READY idempotency + TASK_PAYLOAD_READY/TASK_FAILED; RESULT_READY flow + terminal-outcome blocking + fan-out drain/shutdown gating) | TE | L | TE-3; F3-4 via a narrow stubbed wait-protocol interface. Merged from two Ms: same class, same owner, no independent consumer of either half. Guard: split fan-out drain back out if the diff passes ~1800 LOC | | EP-3 external_process auth + CJ-side HELLO acceptance + session-scoped message enforcement | EP | M | TE-1 (vocabulary). **Owns the auth-model decision**: a per-launch rendezvous id always; a *secret* launch token proven by one-round HMAC on multi-tenant hosts (rendezvous-only + OS isolation may suffice single-tenant). Introduces the proof helpers (TokenScope, compute/verify_hello_proof, combine_nonces) here — reused later by AT-2. Session-scoped message enforcement (accept trainer messages only from the bound session) closes the live IPCAgent any-sender gap; a P0 control, not the P2 attach track | | EX-4 ScriptRunner `execution_mode` param + launch_external_process mapping | EX | M | EX-2/3. Convert ~22 internal recipe call sites in the same PR | -| EX-5 Converter→filter migration (FLARE-2698 bullet 2) | EX | M | EX-3, EP-4. Replace executor-owned ParamsConverters with PT/TF send+receive conversion filters at the client edge (last task-data / first task-result filter); recipes auto-wire per framework only when server format ≠ exchange format; Client API boundary passes through (RAW). Filters delegate to the existing converter classes (no logic rewrite); round-trip state (tensor shapes, excluded entries) rides DXO meta (quantizer pattern), not FLContext — removes the `_ConverterContext` stub. Keep ParamsConverter ABC deprecated + a `ParamsConverterFilter` adapter for custom converter IDs. Wiring the conversion filter makes the CJ a materializing hop (design doc: payload-lifecycle rule); numpy regime excludes bf16 — large models use pytorch end-to-end. Removes params_exchange_format/server_expected_format/converter-id from the surface (already excluded from EX-2's freeze). Transfer type FULL/DIFF stays in model_registry, decided separately | | CC-1 CCWF transfer-declaration plumbing (receiver sets, stage windows, aux passthrough) | CC | M | F3-3. Declaration-only; absent headers preserve today's defaults — its behavior-neutrality is what de-risks the CC track | | CT-8 Session observability (state-transition logs + StatsPoolManager view) | CT | M | EX-2; extends as backends land | @@ -68,6 +67,7 @@ Tracks: **F3** payload layer · **EX** executor/ScriptRunner · **TE** trainer e | PR | Track | Size | Depends | |---|---|---|---| +| EX-5 Converter→filter migration (FLARE-2698 bullet 2) | EX | M | EX-3, EP-4. Replace executor-owned ParamsConverters with PT/TF send+receive conversion filters at the client edge (last task-data / first task-result filter); recipes auto-wire per framework only when server format ≠ exchange format; Client API boundary passes through (RAW). Filters delegate to the existing converter classes (no logic rewrite); round-trip state (tensor shapes, excluded entries) rides DXO meta (quantizer pattern), not FLContext — removes the `_ConverterContext` stub. Keep ParamsConverter ABC deprecated + a `ParamsConverterFilter` adapter for custom converter IDs. Wiring the conversion filter makes the CJ a materializing hop (design doc: payload-lifecycle rule); numpy regime excludes bf16 — large models use pytorch end-to-end. Removes params_exchange_format/server_expected_format/converter-id from the surface (already excluded from EX-2's freeze). Transfer type FULL/DIFF stays in model_registry, decided separately | | CT-2 torchrun 2-rank rank-contract CI tests (CPU/gloo) | CT | M | CT-1, EP-4. EP owner co-reviews | | CT-3 Owner-death (CJ-kill) + payload-lifecycle E2E tests | CT | L | CT-1, F3 track, EP-4. Covers CJ-SIGKILL self-termination + CP reaping; EP owner co-reviews | | CC-2 Swarm onto the transfer contract (remove lazy-ref machinery) | CC | L | CC-1, F3 track, EP-4. Highest-risk PR in the program; maximally isolated revert unit; retires test_lazy_ref_local_aggr / test_msg_root_ttl | @@ -112,18 +112,19 @@ Notes: the one-paragraph "why" for the whole program lives in PR-0 and gets link ## Critical path -F3-1 → F3-4 → TE-4 → TE-5 → EP-4 → CT-1 → CT-3 → release gate. The TE-4 merge removed one review cycle from the path. **EP-4** remains the schedule risk to watch — it's where executor, trainer engine, auth, process runner, and the F3 facade meet; its prerequisites are deliberately extracted to keep it L. +F3-1 → F3-2/F3-3 (parallel) → F3-4 → TE-4 → TE-5 → EP-4 → CT-1 → CT-3 → release gate. F3-2 and F3-3 run in parallel behind F3-1, so the facade's hard dependencies cost the path one M — still inside the 10–14-week floor. The TE-4 merge removed one review cycle from the path. **EP-4** remains the schedule risk to watch — it's where executor, trainer engine, auth, process runner, and the F3 facade meet; its prerequisites are deliberately extracted to keep it L. ## 2.9 cut line (recommendation) -- **P0 — commit for 2.9** (24 PRs): F3 track (4, after folding F3-5 into F3-4) + EX track (4) + TE track (5) + EP track (6) + CT-1/2/3/6/8. Headline: *external_process and in_process on the new Cell stack with an enforceable payload lifecycle, owner-death handling, session-scoped message enforcement, and the config-permission fix*. -- **P1 — strongly target for 2.9** (6 PRs): CC track (5) + the CCWF half of CT-4. Fully severable (SWARM keeps current behavior until CC-2 lands); ship CC-5 flagged experimental. +- **P0 — commit for 2.9** (24 PRs): F3 track (4, after folding F3-5 into F3-4) + EX track (4 — EX-1..4; EX-5 goes to P1) + TE track (5) + EP track (6) + CT-1/2/3/6/8. Headline: *external_process and in_process on the new Cell stack with an enforceable payload lifecycle, owner-death handling, session-scoped message enforcement, and the config-permission fix*. +- **P1 — strongly target for 2.9** (8 PRs): CC track (5) + EX-5 conversion-filter migration (2.9 runs PyTorch end-to-end without it — the numpy regime is what needs the filters) + CT-5 rank-contract example updates (shipped examples must demonstrate the new contract; they do not gate the code cut) + the CCWF half of CT-4. Fully severable (SWARM keeps current behavior until CC-2 lands); ship CC-5 flagged experimental. - **P2 — stretch for 2.9, else 2.9.x/2.10** (4 PRs): AT track (3) + the attach half of CT-4. Attach is the most self-contained step; nothing in P0/P1 depends on it (its reusable auth/proof helpers ship in P0 via EP-3, and session enforcement via EP-3, regardless). - **CT-7 warnings** (1 PR) close 2.9. **Explicitly deferred to 2.10**: the ScriptRunner default flip and any legacy-class removal — ship 2.9 opt-in with warnings; flip after a release of field soak. +- Tier accounting: every one of the 36 PRs has a tier — P0 (24) + P1 unique (7: CC×5, EX-5, CT-5) + P2 unique (3: AT×3) + CT-4 (one PR, halves in P1 and P2) + CT-7 (1) = 36. The P1/P2 headline counts each mention CT-4 once, so the tier lines name 37 items for 36 PRs. ## Effort and staffing -36 PRs. At review-inclusive rates (S ≈ 2 days, M ≈ 1 week, L ≈ 2 weeks) that is ≈ 44 engineer-weeks total; ≈ 34 for P0+P1. With 4–5 engineers owning tracks (F3, TE, EP+EX, CC, AT+CT), full parallelism after Wave 0; calendar floor is the critical path — realistically **10–14 weeks to P0+P1 complete** plus stabilization. If 2.9 code freeze is nearer, cut P2 first, then CC-4/CC-5. +36 PRs. At review-inclusive rates (S ≈ 2 days, M ≈ 1 week, L ≈ 2 weeks) that is ≈ 44 engineer-weeks total; ≈ 36 for P0+P1. With 4–5 engineers owning tracks (F3, TE, EP+EX, CC, AT+CT), full parallelism after Wave 0; calendar floor is the critical path — realistically **10–14 weeks to P0+P1 complete** plus stabilization. If 2.9 code freeze is nearer, cut P2 first, then CC-4/CC-5. ## Guardrails — deliberate non-merges @@ -132,7 +133,7 @@ These pairs are tempting to consolidate and must stay separate: - **EP-1 (0600 fix) with anything**: live credential exposure on today's path; standalone, cherry-pickable, surgical revert. - **EX-3 (in_process backend) + EX-4 (ScriptRunner)**: ScriptRunner is the most-used public entry point (golden exported-config tests, ~22 recipe conversions); a backend parity revert must not drag the public parameter surface. - **Anything into CC-2 (swarm refactor)**: highest-risk PR in the program; must remain a maximally isolated revert unit. -- **F3-3 (budgets) + F3-4 (facade)**: same module, but F3-4 is on the critical path every track waits on; F3-3 serves only CC-1. +- **F3-3 (budgets) + F3-4 (facade)**: same module and strictly sequenced (F3-3 is a hard prerequisite of F3-4), but they stay separate PRs — the budgets land ahead as their own surgical revert unit, and F3-4 is the critical-path PR every track waits on. - **F3-2 (receiver-confirm wire change) must not weld to F3-4's linger/exit-timing policy**: the version-skew change wants early landing and a surgical revert; the linger now lives in F3-4 (former F3-5) but F3-2 still ships isolated ahead of it. - **EP-3 + AT-2**: same CJ-side HELLO/session machinery, but they straddle the P0/P2 boundary — AT-2 consumes EP-3's module instead. - **EP-2 (runner) + EP-5 (CP reaping)**: different processes and blast radii; EP-5's false-positive-reap risk needs an independent revert path. diff --git a/nvflare/client/cell/defs.py b/nvflare/client/cell/defs.py index 57ae47d64f..652e083de5 100644 --- a/nvflare/client/cell/defs.py +++ b/nvflare/client/cell/defs.py @@ -56,6 +56,11 @@ class Topic: # Per task (every round) TASK_READY = "client_api.task_ready" TASK_ACCEPTED = "client_api.task_accepted" + # Sent when the trainer has materialized (or lazily bound) the task payload and hands + # it to user code. Ends the payload-materialization phase: the forward-path heartbeat + # exemption stops here and heartbeats govern the session lease while user code trains + # (design: Heartbeat and Liveness, Revision 2.2). + TASK_PAYLOAD_READY = "client_api.task_payload_ready" TASK_FAILED = "client_api.task_failed" RESULT_READY = "client_api.result_ready" RESULT_ACCEPTED = "client_api.result_accepted" diff --git a/tests/unit_test/client/cell/defs_test.py b/tests/unit_test/client/cell/defs_test.py index b5644a09f2..0b16c0244d 100644 --- a/tests/unit_test/client/cell/defs_test.py +++ b/tests/unit_test/client/cell/defs_test.py @@ -33,6 +33,7 @@ def test_expected_topics_present(self): "HELLO_REJECTED", "TASK_READY", "TASK_ACCEPTED", + "TASK_PAYLOAD_READY", "TASK_FAILED", "RESULT_READY", "RESULT_ACCEPTED", @@ -77,6 +78,7 @@ def test_topic_wire_values(self): "HELLO_REJECTED": "client_api.hello_rejected", "TASK_READY": "client_api.task_ready", "TASK_ACCEPTED": "client_api.task_accepted", + "TASK_PAYLOAD_READY": "client_api.task_payload_ready", "TASK_FAILED": "client_api.task_failed", "RESULT_READY": "client_api.result_ready", "RESULT_ACCEPTED": "client_api.result_accepted",