Skip to content

[CI] Import the DCGM exporter via enroot registry syntax / 用 enroot registry 语法导入 DCGM exporter #2735

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
edwingao28 wants to merge 2 commits into
base: main
Choose a base branch
from
+19 −3
Open

[CI] Import the DCGM exporter via enroot registry syntax / 用 enroot registry 语法导入 DCGM exporter #2735

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
dbea7ea
fix(ci): import the DCGM exporter via enroot registry syntax
edwingao28 Aug 26, 2026
800c874
fix(ci): keep the plain exporter ref where enroot_uri_for_image converts
edwingao28 Aug 26, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion runners/launch_gb200-nv.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -313,8 +313,10 @@ fi

if [[ "$USES_DCGM_POWER" == "1" ]]; then
DCGM_EXPORTER_IMAGE="nvcr.io/nvidia/k8s/dcgm-exporter:4.6.0-4.8.3-distroless"
# enroot resolves bare paths against Docker Hub; nvcr.io pulls need the registry# form
DCGM_EXPORTER_ENROOT_REF="${DCGM_EXPORTER_IMAGE/nvcr.io\//nvcr.io#}"
DCGM_EXPORTER_SQSH="${SQUASH_DIR}/$(echo "$DCGM_EXPORTER_IMAGE" | sed 's/[\/:@#]/_/g').sqsh"
import_squash "$DCGM_EXPORTER_SQSH" "$DCGM_EXPORTER_IMAGE"
import_squash "$DCGM_EXPORTER_SQSH" "$DCGM_EXPORTER_ENROOT_REF"
Comment thread
cursor[bot] marked this conversation as resolved.
Outdated

@chatgpt-codex-connector chatgpt-codex-connector Bot Aug 26, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Pass the original image to the GB200 URI converter

On any GB200 power run where the cached exporter squash is absent or invalid, import_squash passes this argument through enroot_uri_for_image, which already converts nvcr.io/nvidia/... to docker://nvcr.io#nvidia/.... Passing the preconverted value here makes the helper produce docker://nvcr.io#nvidia#k8s/dcgm-exporter:..., with the repository split at the wrong slash, so the exporter import fails and aborts the benchmark; this launcher should continue passing DCGM_EXPORTER_IMAGE.

Useful? React with 👍 / 👎.

Comment on lines +317 to +319

@claude claude Bot Aug 26, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 import_squash's enroot_uri_for_image() (lines 22-64) already rewrites a bare nvcr.io/... image into registry#repo form; passing the pre-rewritten DCGM_EXPORTER_ENROOT_REF (already containing '#') into it double-converts the string.

Extended reasoning...

With DCGM_EXPORTER_ENROOT_REF='nvcr.io#nvidia/k8s/dcgm-exporter:4.6.0-4.8.3-distroless', enroot_uri_for_image splits on the first '/' which now falls inside the repo path, producing registry='nvcr.io#nvidia' and a second '#' before 'k8s', yielding the malformed URI 'docker://nvcr.io#nvidia#k8s/dcgm-exporter:...'. On any cold cache, enroot import fails on GB200 (the exact scenario this PR claims to fix), unlike gb300/h200 which use a plain docker://$image call and are unaffected.

Verification: Severity: normal — regression breaking the exact cold-cache scenario the PR claims to fix. import_squash (launch_gb200-nv.sh line 74) internally calls enroot_uri_for_image on its image arg, and that result is what enroot import uses (line 87). On the BASE branch the call passed the bare DCGM_EXPORTER_IMAGE "nvcr.io/nvidia/k8s/dcgm-exporter:4.6.0-4.8.3-distroless", which enroot_uri_for_image co

test -r "$DCGM_EXPORTER_SQSH" || { echo "Error: DCGM exporter squash not readable: $DCGM_EXPORTER_SQSH" >&2; exit 1; }
unsquashfs -l "$DCGM_EXPORTER_SQSH" > /dev/null || { echo "Error: DCGM exporter squash invalid: $DCGM_EXPORTER_SQSH" >&2; exit 1; }
sha256sum "$DCGM_EXPORTER_SQSH" > "$GITHUB_WORKSPACE/exporter-image.sha256"
Expand Down
4 changes: 3 additions & 1 deletion runners/launch_gb300-nv.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -180,12 +180,14 @@ POWER_SRT_SLURM_PIN="6fc1bed01a0b82dae0088a105c03ce0cfb353443"

if [[ "$USES_DCGM_POWER" == "1" ]]; then
DCGM_EXPORTER_IMAGE="nvcr.io/nvidia/k8s/dcgm-exporter:4.6.0-4.8.3-distroless"
# enroot resolves bare paths against Docker Hub; nvcr.io pulls need the registry# form
DCGM_EXPORTER_ENROOT_REF="${DCGM_EXPORTER_IMAGE/nvcr.io\//nvcr.io#}"
DCGM_EXPORTER_SQSH="/data/home/sa-shared/gharunners/squash/$(echo "$DCGM_EXPORTER_IMAGE" | sed 's/[\/:@#]/_/g').sqsh"
# Note (wenyao): import_squash treats an existing unsquashfs-valid file
# as a cache hit but does not re-validate a fresh import, so check
# explicitly — on a compute node, like the import itself (login node is
# x86, nodes aarch64).
import_squash "$DCGM_EXPORTER_SQSH" "$DCGM_EXPORTER_IMAGE"
import_squash "$DCGM_EXPORTER_SQSH" "$DCGM_EXPORTER_ENROOT_REF"
test -r "$DCGM_EXPORTER_SQSH" || { echo "Error: DCGM exporter squash not readable: $DCGM_EXPORTER_SQSH" >&2; exit 1; }
srun --account="$SLURM_ACCOUNT" --partition="$SLURM_PARTITION" --exclusive --time=30 bash -c "unsquashfs -l \"$DCGM_EXPORTER_SQSH\" > /dev/null" || { echo "Error: DCGM exporter squash invalid: $DCGM_EXPORTER_SQSH" >&2; exit 1; }
sha256sum "$DCGM_EXPORTER_SQSH" > "$GITHUB_WORKSPACE/exporter-image.sha256"
Expand Down
4 changes: 3 additions & 1 deletion runners/launch_h200-dgxc-slurm.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -207,6 +207,8 @@ if [[ "$IS_MULTINODE" == "true" ]]; then

if [[ "$USES_DCGM_POWER" == "1" ]]; then
DCGM_EXPORTER_IMAGE="nvcr.io/nvidia/k8s/dcgm-exporter:4.6.0-4.8.3-distroless"
# enroot resolves bare paths against Docker Hub; nvcr.io pulls need the registry# form
DCGM_EXPORTER_ENROOT_REF="${DCGM_EXPORTER_IMAGE/nvcr.io\//nvcr.io#}"
DCGM_EXPORTER_SQSH="/data/gharunners/containers/$(echo "$DCGM_EXPORTER_IMAGE" | sed 's/[\/:@#]/_/g').sqsh"
if ! unsquashfs -l "$DCGM_EXPORTER_SQSH" >/dev/null 2>&1; then
DCGM_EXPORTER_LOCK="${DCGM_EXPORTER_SQSH}.lock"
Expand All @@ -223,7 +225,7 @@ if [[ "$IS_MULTINODE" == "true" ]]; then
rm -f \"$DCGM_EXPORTER_SQSH\"
export ENROOT_CACHE_PATH=\${HOME}/.cache/enroot
mkdir -p \"\$ENROOT_CACHE_PATH\"
enroot import -o \"$DCGM_EXPORTER_SQSH\" docker://$DCGM_EXPORTER_IMAGE
enroot import -o \"$DCGM_EXPORTER_SQSH\" \"docker://$DCGM_EXPORTER_ENROOT_REF\"
"
fi
test -r "$DCGM_EXPORTER_SQSH" || { echo "Error: DCGM exporter squash is not readable: $DCGM_EXPORTER_SQSH" >&2; exit 1; }
Expand Down
6 changes: 5 additions & 1 deletion utils/test_gb200_power_official_contract.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -87,7 +87,11 @@ def assert_pinned_clone_contract(launcher):

def assert_exporter_provisioning(launcher):
assert f'DCGM_EXPORTER_IMAGE="{EXPORTER_IMAGE}"' in launcher
assert 'import_squash "$DCGM_EXPORTER_SQSH" "$DCGM_EXPORTER_IMAGE"' in launcher
assert (
'DCGM_EXPORTER_ENROOT_REF="${DCGM_EXPORTER_IMAGE/nvcr.io\\//nvcr.io#}"'
in launcher
)
assert 'import_squash "$DCGM_EXPORTER_SQSH" "$DCGM_EXPORTER_ENROOT_REF"' in launcher
assert 'test -r "$DCGM_EXPORTER_SQSH"' in launcher
assert (
'sha256sum "$DCGM_EXPORTER_SQSH" > "$GITHUB_WORKSPACE/exporter-image.sha256"'
Expand Down