[Power] Power measurement roadmap / 功耗测量路线图

[edwingao28]

Links

• Dashboard: https://inferencex.semianalysis.com/inference?i_seq=8k%2F1k&i_metric=y_measuredJPerOutputToken
• H200 exact-pin validation: run 32324249431
• Producer: https://github.com/SemiAnalysisAI/InferenceX
• Consumer: https://github.com/SemiAnalysisAI/InferenceX-app
• Producer fork: edwingao28/srt-slurm#2 at e5c837f

Roadmap

Producer

• Single-node fixed-sequence power, NVIDIA + AMD feat(power): aggregate measured GPU power into agg result JSON #1558 [1/4] [Power] feat: validate single-node GPU energy metrics / 校验单节点 GPU 能耗指标 #2323
• Multinode disaggregated power [2/4] [Power] feat: validate multinode GPU energy metrics / 校验多机 GPU 能耗指标 #2437
• Prefill/decode role watts Emit per-role average power on multinode disaggregated runs / 多节点分离式运行输出每角色平均功率 #2553
• Schema v2 whole-deployment energy semantics [Power] feat: stamp whole-deployment semantics with a schema version / 为功耗指标加盖整机语义版本号 #2599
• Single-node AgentX power [Power] feat: measure GPU power for single-node AgentX runs / 为单机 AgentX 运行接入 GPU 功耗测量 #2601
• GB200/GB300 FP4 DCGM lanes [Power] feat: extend dcgm energy lanes to gb dsv4/qwen3.5 fp4 / 扩展 dcgm 能耗采集到 gb dsv4 与 qwen3.5 fp4 #2507
• AMD multinode power
• Kimi-K3 power evaluation; retain its current vLLM/external-fork clone route until separately qualified

Multinode rollout [1/5]–[5/5]

• [1/5] Long-lived srt-slurm fork prerequisite (merged 2026-08-24) edwingao28/srt-slurm#2
• [2/5] H200 multinode AgentX infrastructure [Power] Wire H200 multinode AgentX power / 接入 H200 多节点 AgentX 功耗链路 #2683 (merged 2026-08-24)
• [3/5] H200 GLM-5.2 + DSV4 recipe rollout [Power] Require GLM-5.2 and add DSV4 H200 telemetry / 强制 GLM-5.2 并新增 DSV4 H200 功耗采集 #2684 (merged 2026-08-24; CI sweep 8/8 = 100%, GLM + DSV4 both publication_valid: true)
• [4/5] GB multinode recipe rollout [Power] Require GB multinode telemetry / 强制启用 GB 多节点功耗采集 #2687 — merged 2026-08-25; validation run 32780220262: GB200 qwen c768 + GB200 dsv4 c16384 + GB300 qwen c5120 all required=True status=complete publication_valid=True; dsv4-GB nightly-pin lanes still blocked by rotted images (pre-existing on main)
• [5/5] B200/B300 multinode rollout [Power] Require B200 and B300 multinode telemetry / 强制启用 B200 和 B300 多节点功耗采集 #2688 — merged 2026-08-26; validation run 32924589757: b200-dgxc + b200-nscale lanes zero failures, b200-nscale-slurm_02 vllm audit required=True complete publication_valid=True (c128/c256 windows valid), B300 both frameworks published valid power; remaining reds attributed to b300-014 enroot perms (cluster ops) and intermittent NIXL; kimik2.6 lanes blocked on runner capacity (also on main)

Multinode rollout validation

• H200 GLM-5.2 strict validation on exact producer e5c837f: run 32324249431 — required: true, publication_valid: true, 32/32 GPUs, schema v2, power_valid: 1
• H200 DSV4 optional recipe validation on producer a1b8c7af: run 32312491367
• H200 CI full validation on e5c837f via [Power] Require GLM-5.2 and add DSV4 H200 telemetry / 强制 GLM-5.2 并新增 DSV4 H200 功耗采集 #2684: run 32767381601 — 8/8 lanes, GLM disagg + DSV4 agg both status=complete publication_valid=True
• Local and CI contract validation for [Power] Wire H200 multinode AgentX power / 接入 H200 多节点 AgentX 功耗链路 #2683, [Power] Require GLM-5.2 and add DSV4 H200 telemetry / 强制 GLM-5.2 并新增 DSV4 H200 功耗采集 #2684, [Power] Require GB multinode telemetry / 强制启用 GB 多节点功耗采集 #2687, and [Power] Require B200 and B300 multinode telemetry / 强制启用 B200 和 B300 多节点功耗采集 #2688
• H200 DSV4 exact-pin refresh on e5c837f (optional; remains required: false)
• H100 SA-Bench dedicated-infrastructure smoke on e5c837f if required for the fork scope
• Native GB hardware validation for [Power] Require GB multinode telemetry / 强制启用 GB 多节点功耗采集 #2687
• Native B200/B300 hardware validation for [Power] Require B200 and B300 multinode telemetry / 强制启用 B200 和 B300 多节点功耗采集 #2688
• After each rollout merge, verify production ingestion and dashboard visibility: schema v2, power_valid scrubbing, whole-deployment J/token semantics, J/query, and role watts where applicable

Power validation series [1/4]–[4/4]

• [1/4] Single-node energy validation [1/4] [Power] feat: validate single-node GPU energy metrics / 校验单节点 GPU 能耗指标 #2323
• [2/4] Multinode energy validation [2/4] [Power] feat: validate multinode GPU energy metrics / 校验多机 GPU 能耗指标 #2437
• [3/4] GB200/GB300 official DCGM lane [3/4] [Power] feat: enable official dcgm energy lane for gb200/gb300 1p1d / 打开 gb200/gb300 官方 1P1D 能耗采集 #2456
• [4/4] AMD MI355X strict monitor lifecycle [4/4] [Power] feat: strict AMD gpu monitor lifecycle for MI355X / MI355X 单节点功率监控严格生命周期 #2494

Upstream srt-slurm

• [1/4] DCGM power artifact data layer #288
• [2/4] Multinode DCGM collection #289 (closed; carried in the fork pin)
• [3/4] Measurement window contract #290
• [4/4] Offline artifact validator + telemetry docs #291 (in review; not a blocker for the long-lived fork)

Dashboard

• Measured energy axes app#720
• Validity guard app#734
• J/query, Wh/query, and %TDP axes app#735
• AI-chart lower-is-better fixes fix(ai-chart): respect metric direction and missing values / 修复 AI 图表指标方向与缺失值 InferenceX-app#798

中文说明

链接

• Dashboard：https://inferencex.semianalysis.com/inference?i_seq=8k%2F1k&i_metric=y_measuredJPerOutputToken
• H200 精确 pin 验证：run 32324249431
• 产出侧：https://github.com/SemiAnalysisAI/InferenceX
• 消费侧：https://github.com/SemiAnalysisAI/InferenceX-app
• 产出侧 fork：edwingao28/srt-slurm#2，固定在不可变 commit e5c837f

产出侧

• NVIDIA + AMD 单节点 fixed-sequence 功耗 feat(power): aggregate measured GPU power into agg result JSON #1558 [1/4] [Power] feat: validate single-node GPU energy metrics / 校验单节点 GPU 能耗指标 #2323
• 多节点分离式功耗 [2/4] [Power] feat: validate multinode GPU energy metrics / 校验多机 GPU 能耗指标 #2437
• 预填充/解码角色功耗 Emit per-role average power on multinode disaggregated runs / 多节点分离式运行输出每角色平均功率 #2553
• Schema v2 全部署能耗语义 [Power] feat: stamp whole-deployment semantics with a schema version / 为功耗指标加盖整机语义版本号 #2599
• 单节点 AgentX 功耗 [Power] feat: measure GPU power for single-node AgentX runs / 为单机 AgentX 运行接入 GPU 功耗测量 #2601
• GB200/GB300 FP4 DCGM lane [Power] feat: extend dcgm energy lanes to gb dsv4/qwen3.5 fp4 / 扩展 dcgm 能耗采集到 gb dsv4 与 qwen3.5 fp4 #2507
• AMD 多节点功耗
• Kimi-K3 功耗评估；在单独完成验证前保留现有 vLLM/外部 fork clone 路径

多节点 rollout [1/5]–[5/5]

• [1/5] 长期使用的 srt-slurm fork 前置 edwingao28/srt-slurm#2
• [2/5] H200 多节点 AgentX 基础设施 [Power] Wire H200 multinode AgentX power / 接入 H200 多节点 AgentX 功耗链路 #2683（依赖 [1/5]）
• [3/5] H200 GLM-5.2 + DSV4 recipe rollout [Power] Require GLM-5.2 and add DSV4 H200 telemetry / 强制 GLM-5.2 并新增 DSV4 H200 功耗采集 #2684（依赖 [2/5]）
• [4/5] GB 多节点 recipe rollout [Power] Require GB multinode telemetry / 强制启用 GB 多节点功耗采集 #2687 — 已合入 2026-08-25，GB200/GB300 三样本验证 publication_valid=True
• [5/5] B200/B300 多节点 rollout [Power] Require B200 and B300 multinode telemetry / 强制启用 B200 和 B300 多节点功耗采集 #2688 — 已合入 2026-08-26，b200 双池零失败、B200/B300 双框架功耗发布验证通过

多节点 rollout 验证

• H200 GLM-5.2 已在精确 producer e5c837f 上完成 strict 验证：run 32324249431 — required: true、publication_valid: true、32/32 GPU、schema v2、power_valid: 1
• H200 DSV4 optional recipe 已在 producer a1b8c7af 上完成验证：run 32312491367
• [Power] Wire H200 multinode AgentX power / 接入 H200 多节点 AgentX 功耗链路 #2683、[Power] Require GLM-5.2 and add DSV4 H200 telemetry / 强制 GLM-5.2 并新增 DSV4 H200 功耗采集 #2684、[Power] Require GB multinode telemetry / 强制启用 GB 多节点功耗采集 #2687、[Power] Require B200 and B300 multinode telemetry / 强制启用 B200 和 B300 多节点功耗采集 #2688 的本地与 CI 契约验证
• 在 e5c837f 上刷新 H200 DSV4 精确 pin 证明（可选；继续保持 required: false）
• 如果 fork 范围需要覆盖 SA-Bench dedicated-infrastructure 路径，在 e5c837f 上运行一次 H100 smoke
• [Power] Require GB multinode telemetry / 强制启用 GB 多节点功耗采集 #2687 的 GB 原生硬件验证
• [Power] Require B200 and B300 multinode telemetry / 强制启用 B200 和 B300 多节点功耗采集 #2688 的 B200/B300 原生硬件验证
• 每个 rollout PR 合入后验证生产摄取与 Dashboard 可见性：schema v2、power_valid 清理、全部署 J/token 语义、J/query，以及适用场景下的 role watts

功耗验证系列 [1/4]–[4/4]

• [1/4] 单节点能耗验证 [1/4] [Power] feat: validate single-node GPU energy metrics / 校验单节点 GPU 能耗指标 #2323
• [2/4] 多节点能耗验证 [2/4] [Power] feat: validate multinode GPU energy metrics / 校验多机 GPU 能耗指标 #2437
• [3/4] GB200/GB300 官方 DCGM lane [3/4] [Power] feat: enable official dcgm energy lane for gb200/gb300 1p1d / 打开 gb200/gb300 官方 1P1D 能耗采集 #2456
• [4/4] AMD MI355X strict monitor 生命周期 [4/4] [Power] feat: strict AMD gpu monitor lifecycle for MI355X / MI355X 单节点功率监控严格生命周期 #2494

上游 srt-slurm

• [1/4] DCGM 功耗产物数据层 #288
• [2/4] 多节点 DCGM 采集 #289（已关闭；由 fork pin 承载）
• [3/4] 测量窗口契约 #290
• [4/4] 离线产物校验器与 telemetry 文档 #291（review 中；不阻塞长期 fork）

Dashboard

• Measured energy 轴 app#720
• Validity guard app#734
• J/query、Wh/query、%TDP 轴 app#735
• AI-chart 的 lower-is-better 修复 fix(ai-chart): respect metric direction and missing values / 修复 AI 图表指标方向与缺失值 InferenceX-app#798