feat(dflash): developer-only online draft tuning — tokens/window ×1.094, decode ×1.105 on gfx1151 (research draft, builds on #819)

[fivetide]

Summary

Draft, research parking PR. Not for merge as-is. This is developer-only online DFlash draft tuning (HIPFIRE_DFLASH_ONLINE_TUNE, off by default).

A small per-request learner re-ranks the draft's top-16 candidates per block row, training on tokens the target has already verified. Verify is unchanged, so every emitted token is the target's.

This is not Path C. It trains no draft weights offline: it is a host-side linear re-ranker learned during the session, about 0.1 ms of host work per cycle.

Builds on #819 (that commit is included here; review that PR first).

spec step (Qwen3.5 chain DFlash, greedy batched LM-head path)
  draft_dflash_block_rank
    draft forward → logits (B-1 rows)
-   argmax_f32_batched → drafts
+   if scratch.online: topk_values_batched_f32_verified_regrid (K=16)
+                      → OnlineDraftTuner::propose (host re-rank) → drafts
  verify (unchanged) → accept → commit
+ DflashSpeculator::step → tuner.observe(position, committed, accepted)   # labels, training
+ configure_request → tuner.reset()     prefill → tuner.seed_prompt(prompt)

generic (llama-family) DFlash chain: same hooks in dflash_generic.rs
  (host top-K via dflash_online::top_k_rows; greedy and temp>0 naive-sampling verify,
   which is exact for any deterministic draft)

Learner features, per candidate:

• rank prior;
• n-gram continuations (orders 1–3) of the proposed chain over the prompt plus the verified text;
• the longest suffix match, plus injection of a missing continuation into slot 15;
• recency;
• depth-scaled logit gap;
• stutter: the candidate equals the chain token 1 or 2 back;
• equals the previous row's top-1, or the next row's top-1/top-2;
• the neighbour rows' (r-1, r+1, r+2) draft probability for the candidate.

The model is linear, score = scale·(l_j − l_0) + w·f_j, trained with Adagrad (lr 0.3, acc0 1) on softmax cross-entropy over the candidates. It trains only on rows a greedy chain reaches, uses one weight set for all row depths, and carries its weights across requests. Text statistics are reset per request.

Results (gfx1151, HIP 7.2)

Online, 20 cold single requests (10 prompts × 2 pairs). Tuned is compared against cached shipping DFlash (tuner unset). Greedy decoding, so tokens/window is deterministic per build.

pair	tokens/window	decode tok/s
Qwen3.5-9B MQ4 + qwen35-9b-dflash-mq4 (strong draft)	×1.044	×1.072
Qwen3-8B MQ4 + qwen3-8b-dflash.hfq (weak, generic path)	×1.146	×1.139
all	×1.094	×1.105

• Sampled decoding (temp 0.7, Qwen3-8B generic path, 10 prompts, hipfire run -n 512): tokens/window ×1.104, faster on 10 of 10.
• By genre: code ×1.05–1.24, prose ×1.00–1.05.
• Prose DFlash still loses to AR on both pairs (about 27 vs 47 tok/s and 11 vs 46 tok/s); tuning does not change that.
• Greedy text: stats mode equals shipping byte for byte. The tuned text can differ at batched-verify near-ties, the same class by which shipping DFlash already differs from AR (example: tuned vs shipping first differs at byte 1140 of 2421, shipping vs AR at byte 54).

Fixtures:

• Qwen3.5-9B qwen3.5-9b.mq4, md5 296092bf1e6a45d78c1acf815eb93366; draft md5 590f35403cd7f1d634945233234a12b7.
• Qwen3-8B qwen3-8b.mq4, md5 8af0eed5b8d2287bb6a33e6c9d03d1f9; draft md5 f4be14d4a6acfabbf4a01f3c79f63dd4.
• Prompts: benchmarks/prompts/online_tune/*.txt (md5s in the perf checkpoint).

Smoke on this branch rebased onto beta 060cadcd3 (daemon md5 ccb5e7a1ab05…): Qwen3.5-9B code_edit_typehints, shipping 796 tok / 80 windows (τ 8.94), tuned 796 tok / 78 windows (τ 9.21). The simulator reproduces ×1.0833 on this build.

Full record (append-only):

• docs/perf-checkpoints/2026-10-05-gfx1151-dflash-online-draft-tuning.md
• docs/perf-checkpoints/2026-10-06-gfx1151-dflash-online-draft-tuning-amendment.md: second pair, replay bias, rejected changes
• docs/perf-checkpoints/2026-10-06-gfx1151-dflash-online-draft-tuning-simulator.md: own-start simulator, final numbers

How to evaluate (read this before changing the learner)

Do not use fixed-start replay of argmax sessions to select changes. It scores a policy at the block starts of a plain-argmax session. A policy that accepts more starts its later blocks on harder positions, which replay cannot see, so it overstated changes by up to about 5%. On one prompt, replay scored τ 1.89 while online gave 1.43.

Use the own-start simulator:

• HIPFIRE_DFLASH_ONLINE_TUNE=sweep drafts a token the target never picks. Every cycle then commits one target token, and the dump (HIPFIRE_DFLASH_ONLINE_DUMP=<file>) holds the draft's top-16 at every position of the greedy text.
• examples/dflash_online_replay.rs --simulate DUMP... walks any policy over those dumps with its own block starts (accept the longest matching prefix, start after the bonus), against argmax on the same text.
• Validation:
  • simulated argmax windows equal shipping windows (80 = 80);
  • per-prompt residual between online and simulation is 2.8% SD;
  • the simulator predicted +2.6% for the last change, and online measured +2.7%.
• --carry-loo estimates a warm daemon: each session runs after the others in one tuner.

# both scripts are developer-only; pause other GPU users; caches under ${DFLASH_TUNE_CACHE:-~/.cache/hipfire-online}
scripts/dflash_online_tune/sim.sh        # primary: sweep dumps (one-time ~2 min/prompt) + --simulate, both pairs, ~10 s after
scripts/dflash_online_tune/online_ab.sh  # confirmation: 20 cold online requests, tuned vs cached shipping (~20 min)
cargo run --release -p hipfire-runtime --example dflash_online_replay -- --simulate --hp lr=0.4 DUMP...   # HP sweeps

• Qwen3-8B pair setup: the scripts create an isolated HIPFIRE_HOME (symlinks to ~/.hipfire) whose config.toml sets [developer] dflash_draft = ".../qwen3-8b-dflash.hfq" and speculation.dflash = "on", because the pair is not registry-managed.
• Held-out set (never used for selection): 8 other committed prompts (fiction_lighthouse, prose_river_short, mixed_code_then_prose, merge_sort_thinking_off, humaneval_0_has_close_elements, lru_cache_pep8_strict, coherence_lloyd_long, bare_factual). The final policy scores ×1.0186 there (×1.0102 for the start-of-segment policy). With --carry-loo it scores q9 ×1.025 and q8 ×1.039.

Knobs

env	meaning
HIPFIRE_DFLASH_ONLINE_TUNE	on re-ranks; stats keeps argmax and prints acceptance-ceiling stats; sweep produces evaluation dumps (always rejected; slow)
HIPFIRE_DFLASH_ONLINE_HP	lr=,acc0=,margin=,inject=,carry=0|1
HIPFIRE_DFLASH_ONLINE_DUMP	binary records (P/L/O/R/S) for the replay tool

Code map

crates/hipfire-runtime/src/dflash_online.rs      # OnlineDraftTuner, Learner, features, dumps, no-peek unit test
crates/hipfire-runtime/examples/dflash_online_replay.rs  # replay / --simulate / --carry-loo
crates/hipfire-runtime/src/dflash.rs             # DflashScratch.online: Option<OnlineDraftTuner>
crates/hipfire-runtime/src/dflash_generic.rs     # generic chain hooks (propose / observe / seed_prompt / reset)
crates/hipfire-arch-qwen35/src/speculative.rs    # greedy batched LM-head hook (+ sweep override)
crates/hipfire-arch-qwen35/src/dflash_spec.rs    # observe / seed_prompt / reset hooks
crates/rdna-compute/src/select_regrid.rs         # topk_values_batched_f32_verified_regrid (gfx1201 + gfx1151)
scripts/dflash_online_tune/{sim,online_ab}.sh    # evaluation harnesses
benchmarks/prompts/online_tune/                  # 10 long-session prompts

topk_values_batched_f32_verified_regrid: the shipping one-block-per-row top-K costs 1349 µs per cycle at 15 × 248320, K=16, which ate the gain on gfx1151. The re-grid kernel takes 63 µs and passes the existing test_select_regrid byte-identity gate on gfx1151 (run with its arch check bypassed; 210 launches, ties/NaN/−inf included). The shipping selector gate (select_regrid_enabled, gfx1201 only) is untouched; only the tuner calls the new entry point.

Tried and rejected

Each was measured with the methods above; details are in the checkpoints.

• Rank-16 LoRA on the LM head over the draft hidden (the original plan):
  • within one request it memorizes (online essay τ 1.10 → 0.94);
  • carried across requests it gives +0.6 pp at about 1 ms/cycle, with a sharply peaked learning rate.
• Per-token bias.
• The target's stale verify argmax past the rejection (right 14% of the time).
• Override gates and EMA weights.
• Off-policy rows and row-weighted loss.
• Shift features, gap interactions and the previous block's view.
• Single and combined feature drops.
• Multi-slot injection.
• Depth-bucketed weights (3 buckets ×1.072 vs 1 weight set ×1.083).
• A held-out-fitted starting prior: +8% in fixed-start replay, +0.4% online (noise).
• Constants (ORDERS, RECENT, MAX_OCC, MAXM) within ±0.2 pp.

Open items / how to pick this up

1. Qwen3.5 sampled (temp>0) path: not wired. The tuner only engages on the greedy batched LM-head path. Configs with temperature > 0 (common in chat) get nothing on Qwen3.5. Generic DFlash at temp 0.7 showed ×1.10, so this is the most valuable next step. Its draft is sampled (q-dependent rejection sampling), so a re-rank must change the draft distribution consistently with the accept rule, or switch that path to a deterministic draft with naive verify as the generic chain does.
2. DFlash2 candidate selector, DDTree, dense-TP mesh ranks: not covered; the tuner is skipped there.
3. Warm-daemon effect: measured only in simulation (--carry-loo). The online harness is one request per fresh daemon.
4. Mismatched-draft case (for example a 3.6 draft on a 3.8 target): blocked. Qwen3.8-27B MQ4XTS on this box emits one token (印) and stops on these prompts with or without speculation. This is a separate issue.
5. serve_harness: cannot drive generic DFlash for the non-registry Qwen3-8B pair. Its serve-path proof reports no request-level DFlash. All evidence here uses hipfire bench (native daemon protocol, noslots).
6. NPU angle (where this started): the learner is about 100 µs of host work, so there is nothing to offload; the LoRA variant that would have needed NPU training did not pay off.
7. Before merge: maintainer buy-in on framing (vs Path C), a decision on whether a host-side dev knob belongs in tree, and claim-scoped serve_harness evidence once item 5 is resolved.

Research history: fork branch fivetide:autoresearch/find-ways-to-improve-performance-in-gpu-npu-inte-20261005 (head 9a5e5e58a), with every experiment as a commit; it also carries the NPU interop work from #818. Dumps and caches on the dev box: ~/.cache/hipfire-online/{sweep,sweep_ho,shipping,hf8home}. Regenerate them with the scripts; they are not portable.

Which surface(s) does this touch?

• kernel: crates/rdna-compute/src/select_regrid.rs (new entry point only; existing kernels unchanged)
• serve: runtime dflash / dflash_generic (hooks dormant unless the env var is set)
• arch crate(s): hipfire-arch-qwen35 (hooks dormant unless the env var is set)
• docs / scripts: perf checkpoints, env-vars, CHANGELOG, scripts/dflash_online_tune/

Test plan

• cargo build --release clean (on beta 060cadcd3 + fix(llama_spec): generic DFlash daemon panic on 1–3 row verify blocks #819)
• cargo test --release -p hipfire-runtime --lib: 974 passed, including training_features_match_proposal_features, a no-peek invariant mutation-checked against two injected leaks
• clippy: no warnings in the new files
• scripts/check-lifecycle.py, scripts/check-env-docs.py pass
• Default path: HIPFIRE_DFLASH_ONLINE_TUNE unset leaves DflashScratch.online = None, so every hook is skipped; stats mode reproduces shipping greedy output byte for byte
• serve_harness battery: blocked on open item 5
• speed-gate.sh: n/a (off by default)

Merge Danger

Door: two-way. Off by default, behind a developer env var, and adds no persistent state or format.

Blast Radius: none at default. When enabled: proposal choice and host time in DFlash chain steps, and one extra top-K launch per cycle on Qwen3.5.