Normalize W4A16 profiler scope grammar and report unique expert count - #1074
Merged
Conversation
roberteg16
marked this pull request as ready for review
August 1, 2026 11:30
Author
|
rocm-scripts counter part PR: https://gitenterprise.xilinx.com/FaaSApps/rocm-scripts/pull/878 |
#1067 made the W4A16 labels self-describing but left three spellings of the same field in circulation: bare `g32` on the dense scopes, `g=32` on the MoE ones, and `gs=`/`sk=` on the AWQ GEMVs. Settle on `key=value` everywhere so a label consumer needs one grammar rather than three. The MoE scope also could not be turned into a bandwidth number. Weight slabs dominate its traffic and the kernel loads one per block, but a trace records expert_ids' length, not its contents -- and several blocks can share an expert, so the block count is only an upper bound (E and top_k give a looser one still). Count the distinct ids and emit them as `experts=`. Reading that count forces a device->host sync, so it is guarded exactly as _moe_gemm_w4a16_scope already guards its own: the helper returns a nullcontext before touching the tensor unless profiling scopes are enabled. The guard has to live in a function rather than inline in the f-string, because Python would otherwise evaluate the argument on every decode step. The sync perturbs the CPU timeline, not the GPU kernel duration measured. Consumer support is in FaaSApps/rocm-scripts#878, which accepts both the old and new spellings, so neither side has to land first. Co-authored-by: Claude Signed-off-by: Robert Esclapez Garcia <robert.garcia@amd.com>
roberteg16
force-pushed
the
rogarcia.w4a16-scope-grammar-experts
branch
from
August 12, 2026 14:25
f960f4f to
faa5824
Compare
Author
|
Current pre-commit is not related to this PR. Merging |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Two producer-side gaps that stop a W4A16 trace from becoming a bandwidth number.
1. Three spellings of one field. #1067 made the labels self-describing, but the group size is written three ways: bare
g32on the dense scopes,g=32on the MoE ones,gs=/sk=on the AWQ GEMVs. Settle onkey=valueeverywhere.hybrid_w4a16.pycoverswvsplitk_int4,hybrid_dequant_w4a16andhybrid_triton_w4a16in one line — they share the_gzsuffix.2. The MoE scope carried no expert count. Weight slabs dominate that kernel's traffic and it loads one per block, but the count cannot be recovered downstream: a trace records
expert_ids' length, not its contents, and several blocks can share an expert. So the block count is only an upper bound, andE/top_ka looser one.fused_moe_wvsplitk_int4now emits the measured value:On the consumer side that is the difference between 5.23 MiB and 33.08 MiB of modelled traffic for a 32-block launch touching 5 distinct experts — 6.3x, in a byte model whose whole job is to be exact.
Test plan
End-to-end capture on
gemma-4-31B-it-AWQ-4bit(decode,--profile --profile-record-shapes --profile-export-trace), with this branch installed:wvsplitk_int4 1x43008x5376 g=32 asymAlso run:
MoE end-to-end on
Qwen3.6-35B-A3B-W4A16-llmcompressor(256 experts, top_k=8, g=128 symmetric, decode with--profile --profile-record-shapes --profile-export-trace):fused_moe_wvsplitk_int4 1x1024x2048 E=256 top_k=8 g=128 sym experts=8experts=8is exactly right: one token, top_k=8, so eight distinct experts[experts=8 measured]and(experts read = 8 of 256, measured), with zero warningshybrid_triton_moe ... g=128 sym, one of the two labels Annotate W4A16 profiler scopes with group size and symmetry #1067 silently broke; it now parses