[kernel] Persistent route-packed (route_block x n_tile) schedule for exl3 fused MoE (sm120)

[brandonmmusic-max]

PR (draft) — A1 route-packed re-tile for the EXL3 fused MoE kernel (sm120)

Target: turboderp-org/exllamav3 (based on v0.0.43).
Independent of the nvfp4 KV work — a standalone sm120 MoE-kernel speedup.

Title

[kernel] Persistent route-packed (route_block × n_tile) schedule for exl3 fused MoE on Blackwell (sm120)

Motivation

The EXL3 fused grouped-MoE kernel underuses consumer Blackwell (sm120):

• quant/exl3_moe_common.cuh: #define MOE_SMS_PER_EXPERT 8 → concurrency 188/8 = 23, 4 SMs
  idle; and smem is hardcoded 90*1024 ("CC 8.6") while sm120 allows ~99 KiB opt-in.
• The fixed 8-SM-per-expert grouping leaves the grouped GEMM tail-heavy and under-tiled.

The change

Port the b12x W4A16 persistent schedule idea onto the exl3 grouped shapes: a persistent
(route_block × n_tile) grid that keeps all 188 SMs busy and split-Ks the grouped GEMM,
replacing the 8-SM-per-expert fixed grouping; use the full sm120 smem opt-in; keep the trellis
codebook dequant (codebook.cuh, exl3_dq.cuh) and the fp16/bf16 mma.m16n8k16 accumulate path
byte-identical (same numerics). Added as a new entry point exl3_moe_fused_retile alongside
the existing exl3_moe/exl3_moe_fused for A/B; opt-in.

Files (diff vs v0.0.43): quant/exl3_moe.cu (+447), quant/exl3_gemm_inner.cuh (+373),
quant/exl3_moe_kernel.cuh (+158), quant/exl3_moe_common.cuh (+46).

Result

Prefill +7–8% on GLM-5.2's 192-expert trellis-3.0 tail (4× RTX PRO 6000 Blackwell, TP4).
Numerics unchanged (trellis dequant + MMA path untouched); shipped and serving coherently since
verdictai/vllm-glm52-tr3-hybrid:v3. The re-tiled path is additive/opt-in, so the existing
exl3_moe/exl3_moe_fused remain byte-identical for fallback.

---

Branch is based on v0.0.43; the diff above is the clean A1 change. It will need a rebase onto v1.0.0 master before merge.

[turboderp]

This would need to be retargeted to the current version (and benchmarked against it). There was already significant changes to the MoE kernel, including a new ticket scheduler (v0.0.43 was round-robin), and also a number of changes to how SMs are allocated.

[brandonmmusic-max]

This would need to be retargeted to the current version (and benchmarked against it). There was already significant changes to the MoE kernel, including a new ticket scheduler (v0.0.43 was round-robin), and also a number of changes to how SMs are allocated.

I will be happy to do that, though it might be a couple of days before I can get to it because I’m working on bringing exl3 based model support to a fork of vllm. Do you have any issue with me marking the PR a draft ?

[turboderp]

Sure, I've converted it to a draft. Let me know if you need any feedback for stuff.