[AgentX] DeepSeek-V4-Pro-0813 llm-d on GB200 - #2719
Conversation
…DEP8) Ports the agentX (llm-manifesto/agentx-mvp) DeepSeek-V4-Pro 1P-EP8/1D-EP8 K8s deployment to InferenceX's SLURM-based llmd-vllm framework on B200 (8 GPUs/node, so each DEP8 role fits on one node vs GB200's 4 GPUs/node needing two). New master-config key dsv4-fp4-b200-llmd-vllm-agentx and recipe carry the agentX EPP scheduling config (disagg-profile-handler's `deciders:` shape, prefix-cache-affinity-filter, token-load-scorer, etc., ported verbatim from a real agentx-mvp results snapshot) and per-role vLLM tuning, dropping the Kubernetes/manifesto-specific Mooncake KV chain in favor of server.sh's existing plain NixlConnector. Wires a self-contained FRAMEWORK==llmd-vllm branch into runners/launch_b200-dgxc.sh (ahead of the srtctl/srt-slurm path used by the other B200 multinode frameworks, which llm-d bypasses entirely), mirroring launch_gb200-nv.sh's pyxis/enroot + slurm_utils.sh helpers pattern. Pins a B200-only EPP/pd-sidecar v0.10.0 binary override (LLMD_BIN_DIR) for the newer EPP plugin shape, isolated from GB200's shared v0.9.0 binaries/path. Co-Authored-By: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Cursor <cursoragent@cursor.com>
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
- dsv4-fp4-b200-llmd-vllm-agentx: drop c256 from the 1P-DEP8/1D-DEP8 conc-list per request. - extract-binaries.sh: support BINARIES_ENV_FILE to source an alternate pins file, so a cluster-scoped router-version bump doesn't require editing the shared binaries.env (whose EPP_FROM_IMAGE/ ROUTING_SIDECAR_IMAGE are plain assignments and can't be overridden via pre-exported env vars). - Add binaries-b200-v0.10.0.env: B200-only v0.10.0 EPP/pd-sidecar pins, linux/amd64 platform (B200 is x86_64, unlike GB200's arm64 default), dedicated LLMD_BIN_DIR. - launch_b200-dgxc.sh: fix the extraction command documented in the LLMD_BIN_DIR comment, which previously relied on the non-overridable env vars above and would have silently extracted v0.9.0 binaries. Co-authored-by: Cursor <cursoragent@cursor.com>
Sibling of the P/D disagg benchmark: one engine per topology does both
prefill and decode (1 node / 8 GPUs each), no P/D KV handoff.
- server.sh/job.slurm: add an aggregated mode (DECODE_NODES=0) - no
decode role, no pd-sidecar, no --kv_transfer_config (there is nothing
to hand KV off to), and the sole engine's rank 0 takes over the
EPP/Envoy/bench coordinator duties normally owned by the decode
leader. Also fixes the underlying DECODE_NODES=0 bugs: job.slurm's
out-of-bounds IPS[] read for DECODE_LEADER_IP, and server.sh emitting
a bogus empty-address decode-0 endpoint in endpoints.yaml.
- New recipes dsv4-fp4-b200-agentx-agg-{tp8,dep8}.yaml: per-role vLLM
tuning ported from agentX's ix-agg-base/agg-tp8 and
ix-agg-dep-base/agg-dep8, plus a minimal single-schedulingProfile EPP
config (no disagg-profile-handler needed - llm-d auto-selects
single-profile-handler for one profile). Needs no EPP router bump
since every plugin used is already in the default v0.9.0 binaries.
- New dsv4_fp4_b200_llmd-vllm-agg.sh wrapper + launch_b200-dgxc.sh now
branches its llmd-vllm wrapper-script choice on $DISAGG instead of
hardcoding the disagg wrapper.
- New dsv4-fp4-b200-llmd-vllm-agentx-agg master-config key (disagg:
false, prefill num-worker=1 / decode num-worker=0 per the
process_result.py / dynamo-vllm aggregated convention). conc-lists
mirror the agentX dspark historical runs (results_ilmarkov-2-ix-agg-
tp8-dspark-v3-long, results_ilmarkov-ix-agg-dep8-dspark-1308-long).
- README.md + perf-changelog.yaml updated.
Co-authored-by: Cursor <cursoragent@cursor.com>
|
Claude finished @ilmarkov's task in 13m 58s —— View job Review of PR #2719 — DeepSeek-V4-Pro llm-d B200 (AgentX)
SummaryFound 3 blocking and 2 warning issues (posted as inline comments):
Doc-consistency nits (no inline comments): many comments reference Positive notes: server.sh's aggregated-mode gating ( |
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
| if [[ "$registry" == "registry-1.docker.io" && "$repository" != */* ]]; then | ||
| repository="library/$repository" | ||
| fi | ||
| printf 'docker://%s#%s:%s\n' "$registry" "$repository" "$digest" |
There was a problem hiding this comment.
🟡 WARNING: The digest branch keeps the tag inside repository, producing a malformed enroot URI for repo:tag@sha256:... images.
Why it matters: For an image like the sibling GB200 llm-d pin quay.io/rh-ee-imarkov/llm-d-nokube-vllm:vllm0.26@sha256:a909..., image_without_digest still contains :vllm0.26, so this prints docker://quay.io#rh-ee-imarkov/llm-d-nokube-vllm:vllm0.26:sha256:a909... — exactly the tag@digest ambiguity this function exists to avoid (per its own header comment). It's latent today only because the new B200 image is tag-only; the moment this image gets digest-pinned like its GB200 counterpart, the import breaks.
Fix:
| if [[ "$registry" == "registry-1.docker.io" && "$repository" != */* ]]; then | |
| repository="library/$repository" | |
| fi | |
| printf 'docker://%s#%s:%s\n' "$registry" "$repository" "$digest" | |
| if [[ "$registry" == "registry-1.docker.io" && "$repository" != */* ]]; then | |
| repository="library/$repository" | |
| fi | |
| # tag@digest form: drop the tag; enroot wants registry#repo:digest. | |
| repository="${repository%%:*}" | |
| printf 'docker://%s#%s:%s\n' "$registry" "$repository" "$digest" |
| # benchmarks/llm-d/extract-binaries.sh | ||
| # before this recipe can actually run; job.slurm's pyxis mount loop is | ||
| # a no-op (falls back to the image's baked-in v0.9.0) until then. | ||
| export LLMD_BIN_DIR="${LLMD_BIN_DIR:-/home/sa-shared/llm-d-bins-v0.10.0}" |
There was a problem hiding this comment.
🟡 WARNING: The v0.10.0 LLMD_BIN_DIR override is applied to every B200 llmd-vllm run, including the aggregated configs that declare router: { version: "0.9.0" }.
Why it matters: This export happens before the DISAGG branch, and job.slurm's mount loop mounts whatever executables it finds there over the image's baked-in binaries. Once /home/sa-shared/llm-d-bins-v0.10.0 is populated (a prerequisite for the disagg recipe to work at all), the dsv4-fp4-b200-llmd-vllm-agentx-agg runs will silently execute the v0.10.0 EPP/pd-sidecar while their master-config metadata, the recipe comments, and the perf-changelog all state they use the default v0.9.0 binaries — the published router version won't match what actually ran.
Fix: Either gate the override on the disagg path (move the export LLMD_BIN_DIR=... inside the DISAGG == true branch, added after the wrapper-selection if), or update the agg key's router: version/comments to 0.10.0 so the metadata matches reality.
There was a problem hiding this comment.
Beyond the inline findings, I checked the ruled-out changelog entry more closely: the Mooncake entry's config-keys (dsv4-fp4-b200-llmd-vllm-agentx) does match its "disagg DEP8 arm" description, since that key names the disagg config (not the -agg aggregated one), so there's no additional config-keys mismatch there.
Extended reasoning...
This run's bug hunt already surfaced five confirmed, non-trivial issues (env-var export bug breaking SLURM submission, append-only violation and invalid PLACEHOLDER pr-links in the byte-sensitive perf-changelog.yaml, a misnamed gb200 prefix on B200-only recipe files, and a malformed enroot URI for tag@ digest images), which on their own are sufficient to require human attention and block approval. I independently re-verified the one ruled-out candidate (the Mooncake entry's config-keys pointing at the disagg key dsv4-fp4-b200-llmd-vllm-agentx versus its "disagg DEP8 arm" description) by reading the surrounding perf-changelog.yaml entries directly, and confirmed the key correctly refers to the disagg config (the aggregated variant is the separate -agg key), so there is no mismatch there. Given the confirmed findings already speak for themselves as inline comments, this defer note is limited to recording that one additional, distinct concern was checked and ruled out rather than restating the existing findings.
Findings marked 🟡 are optional suggestions and need no follow-up push.
| if [[ "$FRAMEWORK" == "llmd-vllm" ]]; then | ||
| # Self-contained: bypasses the srtctl/srt-slurm path entirely (the | ||
| # "Validate framework" guard and the dsv4-only-dynamo-vllm guard | ||
| # further below are srtctl-specific and are never reached from here). | ||
| # MODEL_PATH is already resolved above (the dsv4/fp4 branch); llm-d | ||
| # additionally needs MODEL_NAME (served-model-name), which this file | ||
| # doesn't otherwise set outside the srtctl block - reuse $MODEL (the | ||
| # master-config `model:` field), matching this file's own | ||
| # `export SERVED_MODEL_NAME=$MODEL` convention below. | ||
| if [[ ! ( "$MODEL_PREFIX" == "dsv4" && "$PRECISION" == "fp4" ) ]]; then | ||
| echo "Unsupported MODEL_PREFIX/PRECISION for llmd-vllm on B200: $MODEL_PREFIX/$PRECISION" >&2 | ||
| exit 1 | ||
| fi | ||
| export MODEL_NAME="$MODEL" |
There was a problem hiding this comment.
🔴 The new llmd-vllm block never exports SLURM_PARTITION/SLURM_ACCOUNT before handing off to bash "$BENCH_SCRIPT", which forks submit.sh (a separate process) whose check_env SLURM_ACCOUNT/check_env SLURM_PARTITION require them in the environment; every B200 llmd-vllm job submission fails immediately with "Error: SLURM_ACCOUNT not set". Fix: add export SLURM_PARTITION SLURM_ACCOUNT in the llmd-vllm block, mirroring the sibling tilert branch a few lines below (line 249) which does exactly this before its own subprocess handoff.
Extended reasoning...
SLURM_PARTITION/SLURM_ACCOUNT are set as plain (non-exported) shell vars at the top of the file (lines 5-6) and no workflow sets them as job-level env, so they exist only in this script's local scope. The llmd-vllm branch runs JOB_ID=$(bash "$BENCH_SCRIPT") which execs dsv4_fp4_b200_llmd-vllm-agg.sh/disagg.sh, which runs bash ./submit.sh ... - a brand-new process that only inherits exported vars. submit.sh's check_env SLURM_ACCOUNT / check_env SLURM_PARTITION (both required first) then exit 1 before sbatch ever runs. Contrast with the tilert branch immediately below, which does export SLURM_PARTITION SLURM_ACCOUNT right before its own exec bash "$TILERT_DISAGG" handoff - the same pattern the new block omits.
Verification: normal — the new llmd-vllm block forks a subprocess without exporting the SLURM credentials that subprocess requires. runners/launch_b200-nscale-compat.sh:5-6 set SLURM_PARTITION="${SLURM_PARTITION:-batch_1}" / SLURM_ACCOUNT="${SLURM_ACCOUNT:-benchmark}" as plain (non-exported) vars. The new llmd-vllm branch runs JOB_ID=$(bash "$BENCH_SCRIPT") which execs… | normal — new code…
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=33643687916 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=33643797678 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=33644570874 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=33644970416 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=33649102457 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=33649729754 |
cquil11
changed the title
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=33651246658 |
Take upstream topology (cluster:gb200-nv, 4 GPUs/node, 2 nodes), pinned image SHA, tool-call/reasoning-parser flags, and simplified recipe args. Keep enable_adaptive_verification=true and num_speculative_tokens=5 everywhere per user preference. Resolve perf-changelog ordering conflict by taking upstream keys. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 2 potential issues.
There are 3 total unresolved issues (including 1 from previous review).
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 95586e0. Configure here.
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=33658780455 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=33660753148 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=33679618834 |
2 participants
Changes
cluster:gb200-nv: aggregated TP8/DEP8 on 2 nodes (8 GPUs), or 1P-DEP8/1D-DEP8 on 4 nodes (16 GPUs).Validation
Generated GB200 matrices have 2-node aggregate and 4-node disaggregated allocations. Live sweep validation is in progress; no successful GPU result is claimed yet.
Note
Medium Risk
Touches multinode SLURM orchestration, KV transfer (NIXL/Mooncake), and benchmark result collection; misconfiguration could waste GPU time or produce invalid metrics, but changes are confined to benchmark/CI paths.
Overview
Adds GB200
cluster:gb200-nvAgentX coverage for DeepSeek-V4-Pro-0813 on llm-d/vLLM (DSpark): master keys for aggregated TP8/DEP8 (2 nodes) and disaggregated 1P-DEP8/1D-DEP8 (4 nodes), plus matching agentic recipe YAMLs (Mooncake DRAM offload where declared).The llm-d multinode path gains aggregated mode (
DECODE_NODES=0),recipe.py(DSpark golden AL injection, Mooncake/offload validation, DRAM budget sizing),agentic.sh(discovered vLLM/metricsfor AIPerf), andserver.shupdates (Mooncake MultiConnector, coordinator on rank 0 for agg, stricter required env). GB200 launchers pick agg vs disagg wrappers, stage the 0813 checkpoint, and collect agentic logs; disagg/agg shell wrappers are thinned to delegate tosubmit.sh.submit.sh/job.slurmrequire explicit env (including pyxis vs docker by runner), forward AgentX metadata into containers, and skip legacy router binary mounts whenIS_AGENTIC=1. Docs, changelog, and a unit test for the agentic metrics adapter are included.Reviewed by Cursor Bugbot for commit 11b8af5. Bugbot is set up for automated code reviews on this repo. Configure here.