Skip to content

docs(skills): add GPU performance analysis and inference optimization skills - #6129

Draft
shanmugamr1992 wants to merge 1 commit into
NVIDIA:mainfrom
shanmugamr1992:perf-optimization-skill
Draft

docs(skills): add GPU performance analysis and inference optimization skills#6129
shanmugamr1992 wants to merge 1 commit into
NVIDIA:mainfrom
shanmugamr1992:perf-optimization-skill

Conversation

@shanmugamr1992

Copy link
Copy Markdown
Contributor

Summary

Adds two skills/ guides covering the profile-to-optimization workflow for Megatron-Core inference performance work. Documentation only — no changes to any library or training code.

  • nsight-system-analysis — analyze nsys traces to find the per-iteration gap between two implementations, or break down a single profile. Covers iteration anchoring, GPU busy/idle split (union-of-intervals, not per-stream sums), per-category kernel breakdown, module slicing between GEMM anchors, exposed communication, and single-forward-pass kernel composition. Ships scripts for each step plus a report template and SQL recipes.

  • optimize-inference-siddharth — what to change, in what order, and what never to break when an architecture is slower than vLLM. Encodes the patterns behind Siddharth Singh's (@sidsingh-nvidia) 2026 inference work: CUDA-graph scope and bucket coverage, the inference_optimized MoE stack (NVLS AllGather-V, fused grouped GEMM, shared-expert overlap, pad-row masking), Mamba/SSM scratch sizing, Triton production hygiene, and per-step host overhead — as eleven hard rules with the failure each one prevents.

    It also adds methodology validated on a Qwen3-30B-A3B EP4 campaign that took mcore from 66% to 81% of the vLLM DP4+EP baseline on 4xGB200:

    • Decision gates — quantify a lever's ceiling before writing code. Three worked gates each avoided a multi-week effort, including a hand-written grouped GEMM whose entire ceiling was 1.45x, most of it reachable by tile tuning alone.
    • Competitor-trace differential — is an individual kernel slower, or are there just more of them? Opposite fixes.
    • A/B protocol — session drift reached 1.6%, larger than most remaining wins, so acceptance requires same-session back-to-back arms with non-overlapping distributions rather than a mean delta.
    • nsys failure modes — the trace flag combinations that deadlock finalization on MoE decode under CUDA graphs, and the working substitutes.

The second skill is self-maintaining: references/updating-this-skill.md documents the triggers, routing, quality bar, deletion policy, and revision log, so campaigns extend and prune it rather than letting it go stale.

Test plan

Documentation only; no code paths are touched and nothing is imported by the library or tests.

  • Both SKILL.md frontmatter blocks parse as YAML, with name matching the directory
  • All intra-skill relative links resolve
  • All scripts/*.py byte-compile; taxonomy_template.yml parses
  • license: Apache-2.0 and SPDX headers consistent with the repo, matching the convention of the existing skills/

… skills

Add two skills covering the profile-to-optimization workflow for Megatron-Core
inference performance work:

- nsight-system-analysis: analyze nsys traces to find the per-iteration gap
  between two implementations or break down a single profile. Covers iteration
  anchoring, GPU busy/idle split, per-category kernel breakdown, module slicing
  between GEMM anchors, exposed communication, and single-forward-pass kernel
  composition, with scripts for each step.
- optimize-inference-siddharth: what to change, in what order, and what never to
  break when an architecture is slower than vLLM. Covers CUDA-graph scope and
  bucket coverage, the inference_optimized MoE stack, Triton production hygiene,
  per-step host overhead, decision gates that quantify a lever's ceiling before
  any code is written, competitor-trace differential analysis, and the A/B
  protocol needed to make sub-1% wins falsifiable. The skill is self-maintaining:
  it documents how to extend, correct, and prune itself as campaigns learn more.

Signed-off-by: Shanmugam Ramasamy <111910568+shanmugamr1992@users.noreply.github.com>
@copy-pr-bot

copy-pr-bot Bot commented Jul 29, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant