[AgentX] DeepSeek-v4-pro llm-d NVL72-B200 - #2719
Conversation
…DEP8) Ports the agentX (llm-manifesto/agentx-mvp) DeepSeek-V4-Pro 1P-EP8/1D-EP8 K8s deployment to InferenceX's SLURM-based llmd-vllm framework on B200 (8 GPUs/node, so each DEP8 role fits on one node vs GB200's 4 GPUs/node needing two). New master-config key dsv4-fp4-b200-llmd-vllm-agentx and recipe carry the agentX EPP scheduling config (disagg-profile-handler's `deciders:` shape, prefix-cache-affinity-filter, token-load-scorer, etc., ported verbatim from a real agentx-mvp results snapshot) and per-role vLLM tuning, dropping the Kubernetes/manifesto-specific Mooncake KV chain in favor of server.sh's existing plain NixlConnector. Wires a self-contained FRAMEWORK==llmd-vllm branch into runners/launch_b200-dgxc.sh (ahead of the srtctl/srt-slurm path used by the other B200 multinode frameworks, which llm-d bypasses entirely), mirroring launch_gb200-nv.sh's pyxis/enroot + slurm_utils.sh helpers pattern. Pins a B200-only EPP/pd-sidecar v0.10.0 binary override (LLMD_BIN_DIR) for the newer EPP plugin shape, isolated from GB200's shared v0.9.0 binaries/path. Co-Authored-By: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Cursor <cursoragent@cursor.com>
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
- dsv4-fp4-b200-llmd-vllm-agentx: drop c256 from the 1P-DEP8/1D-DEP8 conc-list per request. - extract-binaries.sh: support BINARIES_ENV_FILE to source an alternate pins file, so a cluster-scoped router-version bump doesn't require editing the shared binaries.env (whose EPP_FROM_IMAGE/ ROUTING_SIDECAR_IMAGE are plain assignments and can't be overridden via pre-exported env vars). - Add binaries-b200-v0.10.0.env: B200-only v0.10.0 EPP/pd-sidecar pins, linux/amd64 platform (B200 is x86_64, unlike GB200's arm64 default), dedicated LLMD_BIN_DIR. - launch_b200-dgxc.sh: fix the extraction command documented in the LLMD_BIN_DIR comment, which previously relied on the non-overridable env vars above and would have silently extracted v0.9.0 binaries. Co-authored-by: Cursor <cursoragent@cursor.com>
Sibling of the P/D disagg benchmark: one engine per topology does both
prefill and decode (1 node / 8 GPUs each), no P/D KV handoff.
- server.sh/job.slurm: add an aggregated mode (DECODE_NODES=0) - no
decode role, no pd-sidecar, no --kv_transfer_config (there is nothing
to hand KV off to), and the sole engine's rank 0 takes over the
EPP/Envoy/bench coordinator duties normally owned by the decode
leader. Also fixes the underlying DECODE_NODES=0 bugs: job.slurm's
out-of-bounds IPS[] read for DECODE_LEADER_IP, and server.sh emitting
a bogus empty-address decode-0 endpoint in endpoints.yaml.
- New recipes dsv4-fp4-b200-agentx-agg-{tp8,dep8}.yaml: per-role vLLM
tuning ported from agentX's ix-agg-base/agg-tp8 and
ix-agg-dep-base/agg-dep8, plus a minimal single-schedulingProfile EPP
config (no disagg-profile-handler needed - llm-d auto-selects
single-profile-handler for one profile). Needs no EPP router bump
since every plugin used is already in the default v0.9.0 binaries.
- New dsv4_fp4_b200_llmd-vllm-agg.sh wrapper + launch_b200-dgxc.sh now
branches its llmd-vllm wrapper-script choice on $DISAGG instead of
hardcoding the disagg wrapper.
- New dsv4-fp4-b200-llmd-vllm-agentx-agg master-config key (disagg:
false, prefill num-worker=1 / decode num-worker=0 per the
process_result.py / dynamo-vllm aggregated convention). conc-lists
mirror the agentX dspark historical runs (results_ilmarkov-2-ix-agg-
tp8-dspark-v3-long, results_ilmarkov-ix-agg-dep8-dspark-1308-long).
- README.md + perf-changelog.yaml updated.
Co-authored-by: Cursor <cursoragent@cursor.com>
1 participant
TBD