Add online-live mode: train drafters from production serving traffic

[sherlockwu]

What

A new online-live training mode: production SGLang servers capture hidden states from real user traffic and push them to SpecForge for drafter training — closing the serve→train loop. The existing driven disagg-online mode (SpecForge originates capture requests from a dataset) is unchanged and the two modes coexist.

Design

The tensor path was already push-based (sink → Mooncake); only the metadata path inverts:

users ──/generate──▶ SGLang (base patch + online-live layer)
                       │ prefill + decode capture rows (one per fed token)
                       │ writer thread: Mooncake put_sample  [tensors]
                       │               POST record → intake  [metadata]
                       ▼
              SpecForge live producer
                CaptureIntakeServer  GET /v1/spec-capture/config   (handshake)
                                     POST /v1/spec-capture/records (dedup/shed/reject)
                LiveIntakeRefSource: validate → SampleRef → adopt → publish
                       ▼
              (unchanged) channel → RefDistributor → trainer → ack → Mooncake free

• Handshake, not flags: the sink GETs its capture document (store id, feature names, passthrough synthesis rules, min/max_num_tokens) from the intake, so SGLang stays algorithm-agnostic. Only two new SGLang flags: --spec-capture-intake-url, --spec-capture-sample-rate.
• Capture covers prefill + generated tokens (rows align with prompt + output[:-1]); capture decision and sample id derive deterministically from the request id so all TP ranks agree.
• Shed-based flow control: user traffic can't be paused, so above the in-flight/byte watermarks the intake answers 429 and the sink drops that capture (removing its keys); the Mooncake pool is the final bound and trainer acks free it. No capture failure ever affects the user's response.
• min_num_tokens: 2 keeps SGLang warmup/probe one-token requests out of training (DFlash-family objectives need two consecutive supervised tokens).
• SGLang changes live in a layered patch patches/sglang/online-live/ on top of the v0.5.14 base patch; scripts/apply_sglang_spec_capture_patch.sh --live applies/reverses it with its own content-aware record.

How to run (3 terminals)

# 0. once: apply the SGLang patch layer
scripts/apply_sglang_spec_capture_patch.sh --live

# 1. SpecForge (live.mooncake in the recipe makes this a managed supervisor
#    owning mooncake_master + producer + consumer)
specforge train --config examples/configs/qwen3-4b-dspark-live.yaml

# 2. capture server (all flags derived from the same config; applies the
#    patch itself unless --skip-patch)
python scripts/online_live/launch_capture_server.py \
    --config examples/configs/qwen3-4b-dspark-live.yaml --cuda 0

# 3. traffic — real users, or the ShareGPT mimic
python scripts/online_live/live_traffic_mimic.py \
    --server-url http://127.0.0.1:30000 \
    --config examples/configs/qwen3-4b-dspark-live.yaml --qps 4 --loop

Manual (non-supervised) launches also work: drop live.mooncake from the config, run your own mooncake_master, and start --role producer / --role consumer separately; the producer prints the exact SGLang flag line at startup.

Validation

• Two full 10k-step qwen3-4b DSpark runs from live traffic on 1 capture GPU + 7 trainer ranks: acceptance accuracy reaches ~0.49–0.54 (position-0 ~0.64–0.69), matching the driven disagg-online curve shape. qps 5: wandb bnbcxd9k; qps 4: wandb y1w0bafs. Over 77k requests: 1 failed request, zero user-visible capture impact; drops were 8.5% shed (by design at qps 5), 0.7% writer-queue, and exactly one warmup request.
• 23 new unit tests (test_live_intake, test_disagg_live_producer — real HTTP → real channel end-to-end, test_sglang_live_capture_patch) plus schema/launch-plan extensions; all suites green (config 58, runtime 632).

v1 limitations (documented in specforge/inference/sglang_patch_inventory.md)

--chunked-prefill-size -1 still required; the capture server cannot itself serve with speculative decoding; retracted requests drop their capture; a sink crash between Mooncake write and key removal leaks hard-pinned orphans until the store restarts; live loss_mask is all-ones.