feat(dsv4): add B200 DeepSeek-V4-Pro dynamo-trt 8k1k disagg throughput recipes / 新增 B200 DeepSeek-V4-Pro dynamo-trt 8k1k 分离式吞吐量配置 - #2721
Conversation
…t recipes Port dsv4-fp4-b200-dynamo-trt from NVIDIA/InferenceMAX PR #177: 16 disaggregated prefill/decode worker-split points (DEP/TEP, with and without MTP3) validated on that repo's B200 pool (16/16, 0 failures). Vendor the srt-slurm recipe YAMLs and EPLB configs, add the master-config entry, and route dynamo-trt for dsv4 in launch_b200-dgxc.sh following the existing kimik2.5 dynamo-trt pattern. Model-path resolution and srtctl install already work for this cluster, so the source PR's NScale-pool-specific workarounds (custom QOS/account overrides, HF checkpoint restaging, uv-cache bypass, ENROOT_REMAP_ROOT patch) are intentionally not carried over. 从 NVIDIA/InferenceMAX PR #177 移植 dsv4-fp4-b200-dynamo-trt：16 个分离式 prefill/decode 工作节点划分配置点(DEP/TEP，含与不含 MTP3)，已在该仓库的 B200 资源池上验证通过(16/16，0 失败)。引入 srt-slurm 配置 YAML 和 EPLB 配置文件，添加 master-config 条目，并按照现有 kimik2.5 dynamo-trt 模式 在 launch_b200-dgxc.sh 中为 dsv4 接入 dynamo-trt 路由。本集群的模型路径 解析和 srtctl 安装流程已可正常工作，因此有意不移植源 PR 中针对 NScale 资源池的专用变通方案(自定义 QOS/账户覆盖、HF 检查点重新暂存、uv 缓存 绕过、ENROOT_REMAP_ROOT 补丁)。 Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
设置 pr-link 为 #2721 Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
… stp/ Move the 16 disagg recipe YAMLs into mtp/ (spec-decoding mtp3) and stp/ (mtp0) subfolders and update the CONFIG_FILE paths in nvidia-master.yaml accordingly. No scenario or search-space changes. 将 16 个分离式配置 YAML 移动到 mtp/(推测解码 mtp3)和 stp/(mtp0)子目录， 并相应更新 nvidia-master.yaml 中的 CONFIG_FILE 路径。场景和搜索空间无变化。 Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…entry Remove the informational RESULTS.md and b200_results.png from the vendored recipe dir (not needed for the sweep) and collapse the mtp/stp directory-split changelog entry back into the single PR entry, since both changes ship in the same PR. 删除已引入配置目录中信息性的 RESULTS.md 和 b200_results.png(sweep 不 需要)，并将 mtp/stp 目录拆分的变更日志条目合并回同一 PR 的单条条目， 因为两处改动同属一个 PR。 Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=32780771012 |
Move the two EPLB moe_load_balancer_*.yaml configs from the shared srt-slurm-recipes/configs/ dir into 8k1k/b200/configs/, alongside the mtp/ and stp/ recipes that reference them, and update the source path in launch_b200-dgxc.sh's cp step accordingly. 将两个 EPLB moe_load_balancer_*.yaml 配置从共享的 srt-slurm-recipes/configs/ 目录移动到 8k1k/b200/configs/ 下，与引用它们的 mtp/ 和 stp/ 配置放在一起， 并相应更新 launch_b200-dgxc.sh 中 cp 步骤的源路径。 Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Pin the srt-slurm checkout for dsv4 dynamo-trt to v1.0.69 (was floating on main), matching the pinned-tag pattern used by the other dynamo-trt/dynamo-sglang multinode branches in this script. 将 dsv4 dynamo-trt 的 srt-slurm 检出固定到 v1.0.69(此前跟随 main 浮动)， 与本脚本中其他 dynamo-trt/dynamo-sglang 多节点分支使用的固定标签模式保持一致。 Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=32781099950 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=32781293131 |
…-trt An off-by-one in the original tail-based extraction from InferenceMAX (NVIDIA/InferenceMAX#177) grabbed two trailing context lines from the prior qwen3.5-fp8-gb300-dynamo-sglang-mtp entry and left them dangling before the new key, outside any mapping. Harmless to the YAML (they parsed as part of the same top-level null-valued mapping quirk) but dead and confusing; remove them. 修复从 NVIDIA/InferenceMAX#177 移植时因逐行提取的差一错误，误将前一个 qwen3.5-fp8-gb300-dynamo-sglang-mtp 条目的两行末尾上下文遗留在新键之前、 未归属任何映射的问题。这两行虽不影响 YAML 解析，但属于死代码且易造成 困惑，现予以移除。 Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=32781378692 |
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
|
|
||
| ep: 16 | ||
| dp-attn: true |
There was a problem hiding this comment.
🔴 New dsv4 dynamo-trt entry's diff leaked stray ep: 16 / dp-attn: true keys into the unrelated, pre-existing glm5.1-fp8-b200-tilert-agentic scenario's decode: mapping, which already defines ep: 1 / dp-attn: false a few lines above (lines 9972-9973).
Extended reasoning...
YAML mappings silently keep the last duplicate key, so PyYAML parses this decode block as ep=16, dp-attn=true instead of the intended ep=1, dp-attn=false — changing the glm5.1 tilert-agentic recipe's expert-parallel/attention-DP settings without any corresponding recipe change, causing that unrelated benchmark to run with the wrong topology or fail once tilert validates the mismatch against its recipe file.
Verification: normal severity. The PR appended (all + lines in the diff) two stray keys immediately after the pre-existing glm5.1 decode block: configs/nvidia-master.yaml lines 9969-9978: ``` decode: num-worker: 1 tp: 8 ep: 1 # 9972 dp-attn: false # 9973 additional-settings: - "DECODE_NODES=1" # 9976
| - "Add B200 DeepSeek-V4-Pro disaggregated dynamo-trtllm 8k1k throughput frontier recipes" | ||
| - "Image: nvcr.io#nvidia/tensorrt-llm/release:1.3.0rc23; 16 prefill/decode worker-split points across DEP/TEP with and without MTP3" | ||
| - "Ported from NVIDIA/InferenceMAX PR #177 (dsv4-fp4-b200-dynamo-trt), adapted to the b200-dgxc runner and existing dsv4 model-path resolution" | ||
| pr-link: https://github.com/NVIDIA/InferenceMAX/pull/177 |
There was a problem hiding this comment.
🟡 New changelog entry's pr-link points to the external source PR (NVIDIA/InferenceMAX#177) instead of this repo's own PR, breaking the established convention of every prior entry.
Extended reasoning...
Every existing pr-link in the file resolves to github.com/SemiAnalysisAI/InferenceX/pull/ (the PR that made the change in this repo); this new entry instead links to github.com/NVIDIA/InferenceMAX/pull/177. Anyone auditing perf history by following pr-link to find the actual InferenceX review/discussion for this benchmark addition lands on an unrelated external repo's PR instead.
Verification: nit. The claim is factually accurate. The new entry (perf-changelog.yaml:6408) sets pr-link: https://github.com/NVIDIA/InferenceMAX/pull/177. A scan of all pr-link: values in the file shows every other populated link points to https://github.com/SemiAnalysisAI/InferenceX/pull/<n> (e.g. the immediately preceding entry uses .../pull/2713); this is the only entry linking to the external NVIDI
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
| enable_block_reuse: false | ||
| free_gpu_memory_fraction: 0.8 | ||
| tokens_per_block: 128 | ||
| max_batch_size: 2 |
There was a problem hiding this comment.
CUDA graph batch exceeds max batch
Medium Severity
Decode cuda_graph_config.batch_sizes includes 4 while max_batch_size is 2. TRT-LLM graph capture for a batch larger than max_batch_size can fail engine setup or waste GPU memory. The sibling 1p2d recipe keeps these values aligned.
Reviewed by Cursor Bugbot for commit 14289a7. Configure here.
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=32781620478 |
The 8k1k MTP recipes pin TLLM_SPEC_DECODE_FORCE_NUM_ACCEPTED_TOKENS in both prefill_environment and decode_environment for throughput measurement. Evals must exercise real speculative decoding, so delete the var from the staged recipes when EVAL_ONLY=true, matching the existing pattern in launch_gb300-nv.sh. 8k1k MTP 配置在 prefill_environment 和 decode_environment 中固定了 TLLM_SPEC_DECODE_FORCE_NUM_ACCEPTED_TOKENS 用于吞吐量测量。评估需要走 真实的推测解码路径，因此在 EVAL_ONLY=true 时从已引入的配置中删除该环境 变量，与 launch_gb300-nv.sh 中的现有模式保持一致。 Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…20260807 Move the dsv4-fp4-b200-dynamo-trt image from nvcr.io#nvidia/tensorrt-llm/release:1.3.0rc23 to 1.3.0rc24 and the dynamo.wheel pin in all 16 8k1k/b200 recipes from 1.4.0.dev20260730 to 1.4.0.dev20260807. Changelog entry updated in place; still one entry for this PR. 将 dsv4-fp4-b200-dynamo-trt 的镜像从 nvcr.io#nvidia/tensorrt-llm/release:1.3.0rc23 升级到 1.3.0rc24，并将全部 16 个 8k1k/b200 配置中的 dynamo.wheel 从 1.4.0.dev20260730 升级到 1.4.0.dev20260807。变更日志条目就地更新，本 PR 仍保持单条条目。 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Resolve the perf-changelog.yaml tail conflict: main's three new entries (#2684, #2701, #2687) keep their original bytes, and the dsv4-fp4-b200-dynamo-trt entry moves to the physical end of the file per the append-only invariant. 解决 perf-changelog.yaml 末尾冲突：main 的三个新条目(#2684、#2701、#2687) 保留原有字节，dsv4-fp4-b200-dynamo-trt 条目按仅追加约束移至文件物理末尾。 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Replace the "Ported from NVIDIA/InferenceMAX PR #177" line with the detail that actually matters for reproducing the numbers: the runner, the srt-slurm pin, and the model-path resolution. 将"Ported from NVIDIA/InferenceMAX PR #177"一行替换为对复现结果真正有用的 信息：运行器、srt-slurm 固定版本以及模型路径解析方式。 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.
There are 2 total unresolved issues (including 1 from previous review).
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 0aafb03. Configure here.
| moe_config: | ||
| backend: TRTLLM | ||
| load_balancer: /configs/moe_load_balancer_ctx_ep4_384.yaml | ||
| moe_expert_parallel_size: 8 |
There was a problem hiding this comment.
Prefill EPLB config EP mismatch
Medium Severity
Prefill moe_config.load_balancer points at moe_load_balancer_ctx_ep4_384.yaml while moe_expert_parallel_size is 8. That EPLB table is built for EP4 slot sharding, so DEP8 prefill places experts with the wrong rank mapping and can unbalance MoE load or OOM a subset of ranks.
Additional Locations (1)
Reviewed by Cursor Bugbot for commit 0aafb03. Configure here.
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=32878857408 |
Retune benchmark.concurrencies in 13 of the 16 8k1k/b200 recipes and mirror them into the dsv4-fp4-b200-dynamo-trt search space so each sweep point requests the concurrency its recipe actually runs. The 1p5d-tep8-b8-mtp3 recipe moves to the multi-value 5x10x45 form, which maps to conc-list [5, 10, 45]. The three unchanged recipes (1p4d-tep8-b4 mtp0/mtp3 and 5p1d-b128-mtp3) keep their values. 在 16 个 8k1k/b200 配置中重新调整了 13 个的 benchmark.concurrencies，并同步 到 dsv4-fp4-b200-dynamo-trt 搜索空间，使每个 sweep 点请求的并发数与其配置 实际运行的一致。1p5d-tep8-b8-mtp3 改用多值形式 5x10x45，对应 conc-list [5, 10, 45]。其余三个未改动的配置(1p4d-tep8-b4 的 mtp0/mtp3 与 5p1d-b128-mtp3)保持原值。 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=32878958006 |
# Conflicts: # perf-changelog.yaml
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=32886832376 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=32906234990 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=32928578692 |
1 participant
Summary
dsv4-fp4-b200-dynamo-trt: 16 disaggregated prefill/decode worker-split points (DEP/TEP, with and without MTP3) for DeepSeek-V4-Pro on B200 with TRT-LLMrelease:1.3.0rc24.moe_load_balancer_*) configs underbenchmarks/multi_node/srt-slurm-recipes/, adds theconfigs/nvidia-master.yamlentry, and routesdynamo-trtfordsv4inrunners/launch_b200-dgxc.shfollowing the existingkimik2.5dynamo-trt pattern.perf-changelog.yamlentry.摘要
dsv4-fp4-b200-dynamo-trt：为 B200 上使用 TRT-LLMrelease:1.3.0rc24的 DeepSeek-V4-Pro 添加 16 个分离式 prefill/decode 工作节点划分配置点（DEP/TEP，含与不含 MTP3）。benchmarks/multi_node/srt-slurm-recipes/下引入 srt-slurm 配置 YAML 和 EPLB（moe_load_balancer_*）配置文件，添加configs/nvidia-master.yaml条目，并按照现有kimik2.5dynamo-trt 模式在runners/launch_b200-dgxc.sh中为dsv4接入dynamo-trt路由。perf-changelog.yaml条目。Test plan
python3.12 utils/matrix_logic/generate_sweep_configs.py full-sweep --config-files configs/nvidia-master.yaml --framework dynamo-trtgenerates 16 rows fordsv4-fp4-b200-dynamo-trtwith no errorsbash -n runners/launch_b200-dgxc.shpassesperf-changelog.yamlparses and the new entry is appended at the physical end, no historical bytes touched🤖 Generated with Claude Code
Note
Low Risk
New benchmark and MoE load-balancer configuration only; no changes to runtime serving or auth paths.
Overview
Adds MoE expert-placement YAML for DeepSeek-V4 on B200 with 384 slots: one file for prefill/context (
moe_load_balancer_ctx_ep4_384) and one for decode/generation (moe_load_balancer_gen_ep8_slots384), each defininginitial_global_assignmentsper rank andnum_slots: 384.Adds two disaggregated TRT-LLM + Dynamo benchmark recipes under
8k1k/b200/mtpthat wire those load balancers, MTP3 speculative decoding, NIXL KV transfer, and sa-bench (8k ISL / 1k OSL): 1p2d (decodemax_batch_size4, concurrency 70) and 1p4d (decode batch 2,max_num_tokens8, concurrency 72).Reviewed by Cursor Bugbot for commit 09804dc. Bugbot is set up for automated code reviews on this repo. Configure here.