[CI] Import the DCGM exporter via enroot registry syntax / 用 enroot registry 语法导入 DCGM exporter - #2735
[CI] Import the DCGM exporter via enroot registry syntax / 用 enroot registry 语法导入 DCGM exporter#2735edwingao28 wants to merge 2 commits into
Conversation
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit dbea7ea. Configure here.
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: dbea7ea4a0
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| DCGM_EXPORTER_ENROOT_REF="${DCGM_EXPORTER_IMAGE/nvcr.io\//nvcr.io#}" | ||
| DCGM_EXPORTER_SQSH="${SQUASH_DIR}/$(echo "$DCGM_EXPORTER_IMAGE" | sed 's/[\/:@#]/_/g').sqsh" | ||
| import_squash "$DCGM_EXPORTER_SQSH" "$DCGM_EXPORTER_IMAGE" | ||
| import_squash "$DCGM_EXPORTER_SQSH" "$DCGM_EXPORTER_ENROOT_REF" |
There was a problem hiding this comment.
On any GB200 power run where the cached exporter squash is absent or invalid, import_squash passes this argument through enroot_uri_for_image, which already converts nvcr.io/nvidia/... to docker://nvcr.io#nvidia/.... Passing the preconverted value here makes the helper produce docker://nvcr.io#nvidia#k8s/dcgm-exporter:..., with the repository split at the wrong slash, so the exporter import fails and aborts the benchmark; this launcher should continue passing DCGM_EXPORTER_IMAGE.
Useful? React with 👍 / 👎.
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
| DCGM_EXPORTER_ENROOT_REF="${DCGM_EXPORTER_IMAGE/nvcr.io\//nvcr.io#}" | ||
| DCGM_EXPORTER_SQSH="${SQUASH_DIR}/$(echo "$DCGM_EXPORTER_IMAGE" | sed 's/[\/:@#]/_/g').sqsh" | ||
| import_squash "$DCGM_EXPORTER_SQSH" "$DCGM_EXPORTER_IMAGE" | ||
| import_squash "$DCGM_EXPORTER_SQSH" "$DCGM_EXPORTER_ENROOT_REF" |
There was a problem hiding this comment.
🔴 import_squash's enroot_uri_for_image() (lines 22-64) already rewrites a bare nvcr.io/... image into registry#repo form; passing the pre-rewritten DCGM_EXPORTER_ENROOT_REF (already containing '#') into it double-converts the string.
Extended reasoning...
With DCGM_EXPORTER_ENROOT_REF='nvcr.io#nvidia/k8s/dcgm-exporter:4.6.0-4.8.3-distroless', enroot_uri_for_image splits on the first '/' which now falls inside the repo path, producing registry='nvcr.io#nvidia' and a second '#' before 'k8s', yielding the malformed URI 'docker://nvcr.io#nvidia#k8s/dcgm-exporter:...'. On any cold cache, enroot import fails on GB200 (the exact scenario this PR claims to fix), unlike gb300/h200 which use a plain docker://$image call and are unaffected.
Verification: Severity: normal — regression breaking the exact cold-cache scenario the PR claims to fix. import_squash (launch_gb200-nv.sh line 74) internally calls enroot_uri_for_image on its image arg, and that result is what enroot import uses (line 87). On the BASE branch the call passed the bare DCGM_EXPORTER_IMAGE "nvcr.io/nvidia/k8s/dcgm-exporter:4.6.0-4.8.3-distroless", which enroot_uri_for_image co
1 participant
What
Import the DCGM exporter image with enroot's
REGISTRY#IMAGEsyntax in the two launchers that pass a baredocker://nvcr.io/...reference straight toenroot import:launch_gb300-nv.shandlaunch_h200-dgxc-slurm.sh.launch_gb200-nv.shis untouched — itsimport_squashroutes every ref throughenroot_uri_for_image, which already emits the registry# form (passing a pre-converted ref there double-parses into a broken URI). The GB200/GB300 contract tests now pin each launcher's correct form.Why
enroot resolves a bare
docker://nvcr.io/nvidia/k8s/dcgm-exporter:...against its default registry (Docker Hub), where the path does not exist — the pull fails with401 Unauthorizedfromregistry-1.docker.io. This exact failure took down every cold-cache exporter import on the b200-nscale cluster (PR #2688 validation, run 32796387229) and was fixed there with thenvcr.io#form; run 32797740765 confirms the converted ref imports cleanly.The GB300/H200 launchers carry the same bare form today and only work because each cluster already holds a cached sqsh — a cache eviction would break the power lanes on those pools. The repo already uses the
nvcr.io#convention elsewhere (launch_h200-dgxc-slurm.shTRT container mapping,launch_h100-dgxc-slurm.sh,launch_gb200-nv.shviaenroot_uri_for_image).The sqsh cache filename is derived separately from
DCGM_EXPORTER_IMAGEand sanitizes/and#identically, so existing caches still hit and non-power lanes are untouched.Validation
bash -non all touched launchersenroot_uri_for_imageexercised locally: plain image →docker://nvcr.io#nvidia/k8s/...(correct); pre-converted ref →docker://nvcr.io#nvidia#k8s/...(broken) — hence the gb200 revertutils:test_gb200_power_official_contract.py,test_gb300_power_official_contract.py,test_h200_power_official_contract.py,test_process_result.py— 80 passed