feat(runtime): add heterogeneous one-producer multi-consumer DFlash fanout

.github/workflows/test.yaml

@@ -63,7 +63,8 @@ jobs:
         run: |
           source sf/bin/activate
           export PYTHONPATH=$PWD
-          bash scripts/apply_sglang_spec_capture_patch.sh
+          bash scripts/apply_sglang_spec_capture_patch.sh \
+            --python "$PWD/sf/bin/python" --apply
           python -c 'import importlib.util, shutil, torch; assert torch.cuda.is_available(), "live capture gate requires CUDA"; assert importlib.util.find_spec("mooncake.store"), "live capture gate requires mooncake.store"; assert shutil.which("mooncake_master") or __import__("os").environ.get("MOONCAKE_MASTER_SERVER_ADDR"), "live capture gate requires mooncake_master"'
           python -m unittest tests.test_runtime.test_server_capture_gate -v
 

examples/configs/README.md

@@ -38,7 +38,8 @@ training, `*-offline.yaml` consumes precomputed features, and
 recipe is disaggregated even when its historical filename only says `online`.
 VLM training is not supported, so the catalog contains text-only recipes.
 
-The `qwen3-8b-dflash-1server-dp7-disaggregated.yaml`,
+The `qwen3-8b-dflash-windowed-fanout.yaml`,
+`qwen3-8b-dflash-1server-dp7-disaggregated.yaml`,
 `qwen3-8b-domino-1server-dp7-disaggregated.yaml`,
 `qwen3-8b-domino-multiserver-disaggregated.yaml`,
 `qwen3.6-27b-dflash-1server-dp2-disaggregated.yaml`, and
@@ -114,6 +115,7 @@ assume the command runs from the repository root.
 | Colocated offline | `qwen3-8b-eagle3-offline.yaml` |
 | External-service online | `qwen3-8b-eagle3-disaggregated.yaml` |
 | Managed-local disaggregated online | `qwen3-8b-domino-multiserver-disaggregated.yaml` |
+| Independent heterogeneous consumers | `qwen3-8b-dflash-windowed-fanout.yaml` |
 | Disaggregated offline | `qwen3-8b-eagle3-offline-disaggregated.yaml` |
 
 The online/offline mode is derived from the selected `data` source, not from
@@ -310,7 +312,7 @@ Managed-local fields:
 
 | Field | Default | What to write |
 | --- | --- | --- |
-| `deployment.disaggregated.managed_local.trainer_cuda_visible_devices` | required | One device token per `nproc_per_node`; trainer and capture devices must not overlap. |
+| `deployment.disaggregated.managed_local.trainer_cuda_visible_devices` | required | One device token per trainer rank, or one per independent `windowed_fanout` consumer; trainer and capture devices must not overlap. |
 | `deployment.disaggregated.managed_local.mooncake` | default object | Owned loopback Mooncake configuration described by the nested fields below. |
 | `deployment.disaggregated.managed_local.capture_servers` | required | One or more owned patched SGLang server definitions. |
 | `deployment.disaggregated.managed_local.shutdown_grace_s` | `30` | Positive graceful process-group shutdown window. |
@@ -336,6 +338,51 @@ endpoints, `store_root`, or `producer_segment_size`. It does not support resume,
 an existing torchrun, or `--node-rank`. All owned ports and GPU assignments must
 be disjoint.
 
+### Independent windowed fanout
+
+`deployment.disaggregated.windowed_fanout` runs one producer and multiple
+single-process consumers from the same YAML. Each child still enters through
+`specforge train`; `launch_plan` selects a consumer with `--consumer-id` and
+projects its loss, block size, anchors, optimizer schedule, seed, checkpoint,
+and output directory onto the canonical trainer configuration.
+
+This topology is intentionally different from data-parallel training. Consumers
+advance independently and may have different step costs. A shared SQLite
+registry records each cursor and bounded capture interest, while Mooncake owns
+the tensor payloads. The producer captures a sample only when a consumer's
+window requests it, reuses a live compatible capture across consumers, and
+reclaims entries outside every legal window.
+
+Required constraints are:
+
+- online DFlash with Mooncake and exactly one prompt epoch;
+- a positive fixed `data.max_prompts`, divisible by batch size times
+  accumulation;
+- `deployment.trainer` fixed at 1x1 because each consumer is its own process;
+- exactly one capture server, shared by every consumer;
+- one managed-local trainer device per consumer, or one explicit
+  `cuda_visible_device` on every external consumer;
+- `max_live_refs`, `max_live_bytes`, and `max_outstanding_per_consumer` large
+  enough for the configured batch and capture reservation.
+
+The main bounds and timing controls are:
+
+| Field | Default | Meaning |
+| --- | --- | --- |
+| `window_lookbehind` / `window_lookahead` | `2` / `40` | Shared default legal window around each consumer cursor. |
+| `max_prefetch_per_consumer` | `8` | Maximum speculative requests inside the lookahead window. |
+| `max_outstanding_per_consumer` | `8` | Hard per-consumer acquired-ref bound; must cover one accumulated optimizer step. |
+| `max_live_refs` / `max_live_bytes` | required byte bound | Global capture capacity shared by all consumers. |
+| `capture_reservation_bytes` | 128 MiB | Bytes reserved transactionally before a capture starts. |
+| `capture_max_sample_bytes` | 128 MiB | Server-side rejection bound for one captured sample; it cannot exceed the transactional reservation. |
+| `capture_batch_size` / `capture_batch_wait_s` | `8` / `0.002` | Producer request batching. |
+| `consumer_prefetch_batches` | `1` | Loader-side prefetch workers for each consumer. |
+
+Each consumer may override `window_lookbehind`, `window_lookahead`, and
+`max_prefetch` in addition to its required training fields. External deployments
+may resume consumers independently with `resume_from`; managed-local runs are
+fresh attempts and reject resume checkpoints.
+
 ### `runtime`: streaming backpressure
 
 This section affects disaggregated streaming producers and is normally omitted

examples/configs/qwen3-8b-dflash-windowed-fanout.yaml

@@ -0,0 +1,98 @@
+model:
+  target_model_path: Qwen/Qwen3-8B
+  draft_model_config: configs/qwen3-8b-dflash.json
+  target_backend: sglang
+  trust_remote_code: true
+  embedding_key: model.embed_tokens.weight
+  torch_dtype: bfloat16
+  mask_token_id: 151669
+  sglang_attention_backend: flashinfer
+  sglang_mem_fraction_static: 0.5
+
+data:
+  train_data_path: ./cache/dataset/perfectblend_qwen3-8b_regen.jsonl
+  max_prompts: 48
+  max_length: 3072
+  chat_template: qwen
+  build_dataset_num_proc: 32
+  cache_dir: ./cache
+
+training:
+  strategy: dflash
+  num_epochs: 1
+  max_steps: 24
+  total_steps: 24
+  batch_size: 2
+  accumulation_steps: 1
+  attention_backend: flex_attention
+  max_grad_norm: 1.0
+  save_interval: 0
+  log_interval: 1
+  dist_timeout: 30
+
+tracking:
+  report_to: none
+
+run_id: qwen3-8b-dflash-windowed-fanout
+output_dir: ./outputs/qwen3-8b-dflash-windowed-fanout
+
+deployment:
+  mode: disaggregated
+  trainer:
+    nnodes: 1
+    nproc_per_node: 1
+  disaggregated:
+    control_dir: ./outputs/qwen3-8b-dflash-windowed-fanout/control
+    backend: mooncake
+    client_buffer_size: 1073741824
+    windowed_fanout:
+      window_lookbehind: 2
+      window_lookahead: 16
+      max_prefetch_per_consumer: 8
+      max_outstanding_per_consumer: 8
+      max_live_refs: 48
+      max_live_bytes: 25769803776
+      capture_reservation_bytes: 536870912
+      capture_max_sample_bytes: 536870912
+      capture_batch_size: 8
+      consumer_prefetch_batches: 1
+      consumers:
+        - consumer_id: dflash-b4
+          seed: 42
+          loss_type: dflash
+          loss_decay_gamma: 7.0
+          dpace_alpha: 0.5
+          draft_block_size: 4
+          num_anchors: 64
+          learning_rate: 0.0006
+          warmup_ratio: 0.04
+        - consumer_id: dflash-b8
+          seed: 43
+          loss_type: dflash
+          loss_decay_gamma: 7.0
+          dpace_alpha: 0.5
+          draft_block_size: 8
+          num_anchors: 128
+          learning_rate: 0.0006
+          warmup_ratio: 0.04
+        - consumer_id: dflash-b16
+          seed: 44
+          loss_type: dflash
+          loss_decay_gamma: 7.0
+          dpace_alpha: 0.5
+          draft_block_size: 16
+          num_anchors: 256
+          learning_rate: 0.0006
+          warmup_ratio: 0.04
+    managed_local:
+      trainer_cuda_visible_devices: ["1", "2", "3"]
+      shutdown_grace_s: 120
+      mooncake:
+        protocol: tcp
+        global_segment_size_bytes: 34359738368
+        local_buffer_size_bytes: 1073741824
+      capture_servers:
+        - port: 30000
+          cuda_visible_devices: ["0"]
+          tp_size: 1
+          mem_fraction_static: 0.5

examples/disagg/README.md

@@ -109,12 +109,20 @@ trainer in one YAML:
 specforge train -c \
   examples/configs/qwen3-8b-dflash-1server-dp7-disaggregated.yaml
 
+specforge train -c \
+  examples/configs/qwen3-8b-dflash-windowed-fanout.yaml
+
 specforge train -c \
   examples/configs/qwen3-8b-domino-1server-dp7-disaggregated.yaml
 ```
 
-These recipes preserve the old DFlash and Domino one-server + DP7
-self-contained topologies. The genuine two-server Domino recipe is:
+The first recipe preserves the DFlash one-server + DP7 topology. The windowed
+recipe instead launches one producer on GPU 0 and three independent DFlash
+consumers on GPUs 1-3. They share compatible captures but keep separate cursors,
+windows, trainer state, and hyperparameters. Its block/anchor pairs are 4/64,
+8/128, and 16/256 so the example does not depend on optional compact-loss or
+custom-kernel optimizations. The Domino recipe preserves its
+one-server + DP7 topology. The genuine two-server Domino recipe is:
 
 ```bash
 specforge train -c \