Skip to content

Normalize W4A16 profiler scope grammar and report unique expert count - #1074

Merged
roberteg16 merged 1 commit into
gfx11from
rogarcia.w4a16-scope-grammar-experts
Aug 12, 2026
Merged

Normalize W4A16 profiler scope grammar and report unique expert count#1074
roberteg16 merged 1 commit into
gfx11from
rogarcia.w4a16-scope-grammar-experts

Conversation

@roberteg16

@roberteg16 roberteg16 commented Jul 31, 2026

Copy link
Copy Markdown

Summary

Two producer-side gaps that stop a W4A16 trace from becoming a bandwidth number.

1. Three spellings of one field. #1067 made the labels self-describing, but the group size is written three ways: bare g32 on the dense scopes, g=32 on the MoE ones, gs=/sk= on the AWQ GEMVs. Settle on key=value everywhere.

- wvsplitk_int4 1x43008x5376 g32 asym
+ wvsplitk_int4 1x43008x5376 g=32 asym

- awq_gemv_hip 5376x4096 gs=128 sk=2
+ awq_gemv_hip 5376x4096 g=128 split_k=2

hybrid_w4a16.py covers wvsplitk_int4, hybrid_dequant_w4a16 and hybrid_triton_w4a16 in one line — they share the _gz suffix.

2. The MoE scope carried no expert count. Weight slabs dominate that kernel's traffic and it loads one per block, but the count cannot be recovered downstream: a trace records expert_ids' length, not its contents, and several blocks can share an expert. So the block count is only an upper bound, and E/top_k a looser one. fused_moe_wvsplitk_int4 now emits the measured value:

fused_moe_wvsplitk_int4 4x1024x2048 E=256 top_k=8 g=128 sym experts=5

On the consumer side that is the difference between 5.23 MiB and 33.08 MiB of modelled traffic for a 32-block launch touching 5 distinct experts — 6.3x, in a byte model whose whole job is to be exact.

Test plan

End-to-end capture on gemma-4-31B-it-AWQ-4bit (decode, --profile --profile-record-shapes --profile-export-trace), with this branch installed:

  • Labels come out in the new grammar: wvsplitk_int4 1x43008x5376 g=32 asym
  • The roofline consumer reads them with zero warnings
  • Per-shape bandwidth 188-204 GiB/s (81.8-88.8% of peak), matching the pre-change capture's 188.6-202.6 GiB/s within run-to-run noise

Also run:

MoE end-to-end on Qwen3.6-35B-A3B-W4A16-llmcompressor (256 experts, top_k=8, g=128 symmetric, decode with --profile --profile-record-shapes --profile-export-trace):

  • The scope emits the measured count in a real run:
    fused_moe_wvsplitk_int4 1x1024x2048 E=256 top_k=8 g=128 sym experts=8
  • experts=8 is exactly right: one token, top_k=8, so eight distinct experts
  • The window attaches to the right op — the roofline reports
    [experts=8 measured] and (experts read = 8 of 256, measured), with zero warnings
  • The same trace also exercises hybrid_triton_moe ... g=128 sym, one of the two labels Annotate W4A16 profiler scopes with group size and symmetry #1067 silently broke; it now parses

@roberteg16
roberteg16 marked this pull request as ready for review August 1, 2026 11:30
@roberteg16
roberteg16 requested a review from jimw567 August 1, 2026 11:30
@roberteg16

Copy link
Copy Markdown
Author

rocm-scripts counter part PR: https://gitenterprise.xilinx.com/FaaSApps/rocm-scripts/pull/878

#1067 made the W4A16 labels self-describing but left three spellings of the
same field in circulation: bare `g32` on the dense scopes, `g=32` on the MoE
ones, and `gs=`/`sk=` on the AWQ GEMVs. Settle on `key=value` everywhere so a
label consumer needs one grammar rather than three.

The MoE scope also could not be turned into a bandwidth number. Weight slabs
dominate its traffic and the kernel loads one per block, but a trace records
expert_ids' length, not its contents -- and several blocks can share an
expert, so the block count is only an upper bound (E and top_k give a looser
one still). Count the distinct ids and emit them as `experts=`.

Reading that count forces a device->host sync, so it is guarded exactly as
_moe_gemm_w4a16_scope already guards its own: the helper returns a
nullcontext before touching the tensor unless profiling scopes are enabled.
The guard has to live in a function rather than inline in the f-string,
because Python would otherwise evaluate the argument on every decode step.
The sync perturbs the CPU timeline, not the GPU kernel duration measured.

Consumer support is in FaaSApps/rocm-scripts#878, which accepts both the old
and new spellings, so neither side has to land first.

Co-authored-by: Claude
Signed-off-by: Robert Esclapez Garcia <robert.garcia@amd.com>
@roberteg16
roberteg16 requested a review from mgehre-amd August 12, 2026 14:21
@roberteg16
roberteg16 force-pushed the rogarcia.w4a16-scope-grammar-experts branch from f960f4f to faa5824 Compare August 12, 2026 14:25

@mgehre-amd mgehre-amd left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

@roberteg16

Copy link
Copy Markdown
Author

Current pre-commit is not related to this PR. Merging

@roberteg16
roberteg16 merged commit da24cbd into gfx11 Aug 12, 2026
3 of 5 checks passed
@roberteg16
roberteg16 deleted the rogarcia.w4a16-scope-grammar-experts branch August 28, 2026 18:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants