perf(ggml/hip): optimize ROCMFPX fp3/fp2 MMVQ on RDNA (perm-LUT + VDR=4 + fp2)

[DeanoC]

Summary

Single squashed PR (supersedes #19 + #20 — combined for easier review) optimizing the ROCMFPX single-token MMVQ path (batch-1 decode + DSpark spec seed) for the DeepSeek-V4-Flash expert projections on RDNA/gfx1151. All changes are HIP-only and gated to the ROCMFPX fp3/fp2 types; all other quant types and non-HIP builds are untouched.

Depends on (and measured against)

Stacks on two open upstream PRs — both prerequisites and the benchmark base:

• feat(ds4): add DSpark speculative decoding (+27% on Strix Halo) Luce-Org/lucebox#496 — DSpark speculative runtime (the decode path the benchmark exercises; head df6f907, an exact ancestor of the base).
• perf(ggml/hip): add batched ROCmFPX MMQ Luce-Org/lucebox#513 — batched ROCmFPX MMQ (head ad73cb8, an exact ancestor of the base).

The PR base branch base/ds4-mmq-spec-b9e5804 is exactly #496 merged with #513; only the kernel changes below differ from it.

Changes

fp3 (Q3_0_ROCMFPX)

1. Vectorized code decode (vecdotq.cuh) — decode 4 fp3 codes with a single v_perm_b32 over an 8-byte LUT (rocmfpx_pack4_fp3_bits12_vec_cuda) instead of 4 scalar decodes + make_char4; extract the code window with one dword-splice instead of a per-bit OR loop. Bit-identical.
2. VDR=4 half-block-aligned vec_dot — at VDR=4 the MMVQ launcher dispatches iqs ∈ {0,4}, so each vec_dot covers exactly one 16-weight half-block (single scale, no straddle branch); the 48-bit weight window is spliced out in one 64-bit read and fed through two independent dp4a chains. Doubles work-per-lane, halves lanes-per-block and the K-loop trip count. This is the one non-bit-identical change (see Correctness).
3. ds_swizzle warp reduction (mmvq.cu) — reduce the fp3 partial sums with a bitmask-mode ds_swizzle butterfly instead of the ds_bpermute lowering HIP emits for warp_reduce_sum; identical {16,8,4,2,1} order and float adds. Bit-identical.

fp2 (Q2_0_ROCMFP2)
4. Two independent dp4a chains (integer accumulation is associative → bit-identical) and the ds_swizzle reduction extended from fp3 to fp2 (shared helper warp_reduce_sum_rocmfpx_dsswizzle). Bit-identical.

Benchmark

• Hardware: AMD Ryzen AI Max (gfx1151 / Strix Halo, RDNA3.5 APU), ROCm 7.2.4, unified memory.
• Model: DeepSeek-V4-Flash, native ROCMFPX GGUF (ROCMFP2 experts / fp3 down-proj / ROCMFP4 attention).
• Workload: DSpark speculative decode (--verify-width 4), greedy (temperature 0), fixed 160-token prompt.
• Clock control: mclk pinned to 1000 MHz (rocm-smi --setmclk 2) — the BW-dominant clock, effectively fixed. sclk is not lockable on this APU (the SMU firmware ignores perf-determinism, DPM-mask, and sclk-ceiling controls; it floats 600↔2800 MHz under load). So each config runs 2 full-length warm-ups to reach steady clock, then 7 measured reps, and we report the achieved sclk per config so every comparison is matched-clock (approximate fixed clock ~1850 MHz). Rejected any rep whose sclk dipped (contention).
• Base: #496 merged with #513 (base/ds4-mmq-spec-b9e5804, this PR's parent).

Cumulative ladder — each adjacent pair was A/B'd in the same session at matched clock:

stage	decode tok/s (median, n=7)	accept_rate	achieved sclk (MHz)
base b9e5804 (Luce-Org#496+Luce-Org#513)	21.7 (± 0.14)	0.78	1786
+ perm-LUT / fold / ds_swizzle (fp3)	24.3 (± 0.07)	0.78	1843
+ fp3 VDR=4 + fp2 (this PR completes)	26.6 (± 0.06)	0.79	1863

• Rung 1 (base → perm-LUT/fold/ds_swizzle): +12.0% median, +11.0% matched-clock (at ~1855 MHz: base 21.9, PR 24.3). Reproduced across two runs (+12.6%, +12.0%).
• Rung 2 (→ VDR=4 + fp2): +9.0%, matched-clock — measured head-to-head this session as 24.4 tok/s @ 1861 MHz vs 26.6 tok/s @ 1863 MHz (both σ ≈ 0.05, n=7); the slightly lower sclk on the faster build means the clock-normalized gain is marginally higher.
• Total vs base: +22.6% (21.7 → 26.6), accept_rate 0.78 → 0.79.

fp3-isolated kernel time — type-104 (fp3) vs the unchanged type-101 (fp4-fast) as a same-run, clock-independent control (rocprofv3 --kernel-trace): −18.5% (ratio 0.379 → 0.309).

Correctness

Everything except the VDR=4 step is byte-identical to base — perm-LUT decode, epilogue fold, ds_swizzle (fp3 & fp2), and the entire fp2 change — verified by SHA-256 match on greedy output.

The VDR=4 fp3 rewrite is not byte-identical: regrouping the per-lane accumulation reassociates the cross-lane float reduction (≤ ~1 ULP per block, far below the argmax margin; the integer dp4a sums stay exact). Validated by:

• greedy math (= 630 L) and 3-pet logic puzzle correct;
• structured prompts (JSON / planet facts / number spelling) byte-identical to base;
• spec accept_rate preserved (0.78 → 0.79);
• 8-prompt diverse greedy quality suite (code, algorithms, exact arithmetic, logic, long-form, JSON, factual recall, repetition) — no regressions.

If a byte-identical guarantee is preferred, the VDR=4 hunk is the only non-byte-identical part and can be dropped (keeps the +12% byte-identical rung); happy to split on request.

Reproducing the correctness checks (e2e)

Build the base and this PR (gfx1151), then run the same GGUF greedy on both and compare.

# 1. Build both (from server/), for each of base/ds4-mmq-spec-b9e5804 and this branch:
cmake -S server -B build-gfx1151 -DCMAKE_BUILD_TYPE=Release \
  -DCMAKE_HIP_ARCHITECTURES=gfx1151 -DCMAKE_HIP_FLAGS=-DDFLASH_WAVE_SIZE=32 \
  -DDFLASH27B_GPU_BACKEND=hip -DGGML_HIP=ON -DGGML_HIP_MMQ_MFMA=ON \
  -DGGML_HIP_NO_VMM=ON -DGGML_HIP_GRAPHS=OFF -DGGML_HIP_ROCWMMA_FATTN=OFF
cmake --build build-gfx1151 --target dflash_server -j$(nproc)

# 2. Launch each build (DSpark spec decode; env vars, not flags):
DFLASH_DS4_SPEC=1 DFLASH_DS4_DRAFT=/path/to/draft.gguf DFLASH_DS4_FUSED_VERIFY=1 DFLASH_DS4_TIMING=1 \
  ./build-gfx1151/dflash_server /path/to/main.gguf --target-device hip:0 --draft-device hip:0 \
  --host 127.0.0.1 --port 8080 --max-ctx 4096 --default-max-tokens 400 --verify-width 4
# log must show "DSpark spec-decode ENABLED"

# 3. Per prompt, greedy (temperature 0), SHA-256 the completion; compare base vs PR:
PROMPT='List the eight planets of our solar system in order from the Sun, one per line, numbered 1 to 8.'
curl -s localhost:8080/v1/chat/completions -H 'Content-Type: application/json' \
  -d "{\"model\":\"dflash\",\"messages\":[{\"role\":\"user\",\"content\":\"$PROMPT\"}],\"max_tokens\":400,\"temperature\":0}" \
  | jq -r '.choices[0].message.content' | sha256sum

Expected:

• Byte-identical set (structured/short outputs — JSON, factual lists, number spelling): SHA-256 matches base exactly. This covers the perm-LUT, fold, ds_swizzle, and all of fp2.
• VDR=4-affected set (longer free-form generations): output may differ by a late token (≤1-ULP float reassociation) but must stay correct — e.g. the arithmetic word problem still yields 630, the 3-pet logic puzzle still solves to Ana=dog/Ben=fish/Cara=cat.
• accept_rate from the DSpark decode: … accept_rate= server log line is preserved (0.78 → 0.79) — a degraded kernel would drop draft acceptance.

We ran this over an 8-prompt suite (code, algorithms, exact arithmetic, logic, long-form, JSON, factual recall, repetition); the byte-identical set matched and the VDR=4 set stayed correct with accept_rate held.

Scope

HIP + GGML_TYPE_Q3_0_ROCMFPX / GGML_TYPE_Q2_0_ROCMFP2 only, gated by #if defined(GGML_USE_HIP) plus constexpr type checks and a width == 32 wave32 guard (non-HIP, CUDA, and wave64/CDNA fall back to the generic kernel). Apache-2.0. For upstreaming to Luce-Org/lucebox this depends on Luce-Org#496 + Luce-Org#513 landing there first.

[DeanoC]

Thanks @pramodith — addressed both:

Kernel-level correctness tests (ec48639): added host unit tests in ggml/rocmfpx/test_rocmfpx.c, wired as a ctest target (rocmfpx_unit) that's built and run in the hosted CI job. They pin the decode contract the vec_dot_rocmfpx_fp3/fp2_q8_1 kernels rely on:

• code→value tables — fp3 {0,1,2,4,0,-1,-2,-4}, fp2 {-1,0,1,2}: the mapping the device perm-LUT encodes, already tied to it by a compile-time static_assert in vecdotq.cuh.
• decode contract — reconstruct each weight from unpacked code → value × half-block scale using the kernels' bit-unpacking, and assert bit-exact against rocmfpx_dequantize_row_*. This pins the bit order, the code table, and the e[i >= QK/2] half-block scale split all at once.
• dot structure — db · Σ_half( e_half · Σ_i code·q8 ) == the float reference dot (the VDR=4 fp3 / two-chain fp2 accumulation), rel err < 1e-5.

It's host-only (compiles rocmfpx.c directly, no GPU), so it runs on the GitHub-hosted job — added to the ci.yml build-target list and the ctest -R filter.

One note: a full on-GPU GPU-vs-CPU-reference check via test-backend-ops would be even stronger (both types already have to_float + from_float_ref, so an NMSE MUL_MAT check would exercise the actual mul_mat_vec_q HIP kernel). That harness isn't vendored in this fork's tree, so I left it out here — happy to file it as a follow-up if you'd like the on-device coverage.

E2E correctness commands: added a Reproducing the correctness checks (e2e) section to the PR description — build base + this PR, run greedy (temp 0) DSpark spec decode on both, SHA-256 compare the byte-identical prompts, and verify the VDR=4 set stays correct (arithmetic → 630, logic puzzle solved) with accept_rate held.

[pramodith]

Nice wins with bit arithmetic!