When trying to launch a node with one of our recent CCCL containers cccl-devcontainers:cpp-llvm21-cuda99.8 from gitlab I get the following error.
ggonidelis@computelab-sc-01-frontend-01:/home/scratch.ggonidelis_sw$ salloc --time=1:00:00 --partition="rtx-3090@ts3/x11spitf/1gpu-16cpu-64gb" --nodelist=x11-0436 --gres=gpu:1 srun --pty --container-image=gitlab-master.nvidia.com/cccl/cccl-devcontainers:cpp-llvm21-cuda99.8 --container-mounts=/home/scratch.ggonidelis_sw:/src bash
salloc: Granted job allocation 2837533
salloc: Nodes x11-0436 are ready for job
pyxis: importing docker image: gitlab-master.nvidia.com/cccl/cccl-devcontainers:cpp-llvm21-cuda99.8
[2026-06-26T16:44:27.587] error: pyxis: child 133018 failed with error code: 18
[2026-06-26T16:44:27.587] error: pyxis: failed to import docker image
[2026-06-26T16:44:27.587] error: pyxis: printing enroot log file:
[2026-06-26T16:44:27.587] error: pyxis: [INFO] Querying registry for permission grant
[2026-06-26T16:44:27.587] error: pyxis: [INFO] Authenticating with user: ggonidelis
[2026-06-26T16:44:27.587] error: pyxis: [INFO] Using credentials from file: /home/ggonidelis/.config/enroot/.credentials
[2026-06-26T16:44:27.588] error: pyxis: [INFO] Authentication succeeded
[2026-06-26T16:44:27.588] error: pyxis: [INFO] Fetching image manifest list
[2026-06-26T16:44:27.588] error: pyxis: [INFO] Fetching image manifest
[2026-06-26T16:44:27.588] error: pyxis: [INFO] Downloading 25 missing layers...
[2026-06-26T16:44:27.588] error: pyxis: [INFO] Extracting image layers...
[2026-06-26T16:44:27.588] error: pyxis: tar: This does not look like a tar archive
[2026-06-26T16:44:27.588] error: pyxis: tar: Skipping to next header
[2026-06-26T16:44:27.588] error: pyxis: tar: Exiting with failure status due to previous errors
[2026-06-26T16:44:27.588] error: pyxis: tar: This does not look like a tar archive
[2026-06-26T16:44:27.588] error: pyxis: tar: Skipping to next header
[2026-06-26T16:44:27.588] error: pyxis: tar: Exiting with failure status due to previous errors
[2026-06-26T16:44:27.588] error: pyxis: tar: This does not look like a tar archive
[2026-06-26T16:44:27.588] error: pyxis: tar: Skipping to next header
[2026-06-26T16:44:27.588] error: pyxis: tar: Exiting with failure status due to previous errors
[2026-06-26T16:44:27.588] error: pyxis: tar: This does not look like a tar archive
[2026-06-26T16:44:27.588] error: pyxis: tar: Skipping to next header
[2026-06-26T16:44:27.588] error: pyxis: tar: Exiting with failure status due to previous errors
[2026-06-26T16:44:27.588] error: pyxis: tar: This does not look like a tar archive
[2026-06-26T16:44:27.588] error: pyxis: tar: Skipping to next header
[2026-06-26T16:44:27.588] error: pyxis: tar: Exiting with failure status due to previous errors
[2026-06-26T16:44:27.588] error: pyxis: tar: This does not look like a tar archive
[2026-06-26T16:44:27.588] error: pyxis: tar: Skipping to next header
[2026-06-26T16:44:27.588] error: pyxis: tar: Exiting with failure status due to previous errors
[2026-06-26T16:44:27.588] error: pyxis: tar: This does not look like a tar archive
[2026-06-26T16:44:27.588] error: pyxis: tar: Skipping to next header
[2026-06-26T16:44:27.588] error: pyxis: tar: Exiting with failure status due to previous errors
[2026-06-26T16:44:27.588] error: pyxis: tar: This does not look like a tar archive
[2026-06-26T16:44:27.588] error: pyxis: tar: Skipping to next header
[2026-06-26T16:44:27.588] error: pyxis: tar: Exiting with failure status due to previous errors
[2026-06-26T16:44:27.588] error: pyxis: tar: This does not look like a tar archive
[2026-06-26T16:44:27.588] error: pyxis: tar: Skipping to next header
[2026-06-26T16:44:27.588] error: pyxis: tar: Exiting with failure status due to previous errors
[2026-06-26T16:44:27.588] error: pyxis: tar: This does not look like a tar archive
[2026-06-26T16:44:27.588] error: pyxis: tar: Skipping to next header
[2026-06-26T16:44:27.588] error: pyxis: tar: Exiting with failure status due to previous errors
[2026-06-26T16:44:27.588] error: pyxis: tar: This does not look like a tar archive
[2026-06-26T16:44:27.588] error: pyxis: tar: Skipping to next header
[2026-06-26T16:44:27.588] error: pyxis: tar: Exiting with failure status due to previous errors
[2026-06-26T16:44:27.588] error: pyxis: tar: This does not look like a tar archive
[2026-06-26T16:44:27.588] error: pyxis: tar: Skipping to next header
[2026-06-26T16:44:27.588] error: pyxis: tar: Exiting with failure status due to previous errors
[2026-06-26T16:44:27.588] error: pyxis: tar: This does not look like a tar archive
[2026-06-26T16:44:27.588] error: pyxis: tar: Skipping to next header
[2026-06-26T16:44:27.588] error: pyxis: tar: Exiting with failure status due to previous errors
[2026-06-26T16:44:27.588] error: pyxis: tar: This does not look like a tar archive
[2026-06-26T16:44:27.588] error: pyxis: tar: Skipping to next header
[2026-06-26T16:44:27.588] error: pyxis: tar: Exiting with failure status due to previous errors
[2026-06-26T16:44:27.588] error: pyxis: tar: This does not look like a tar archive
[2026-06-26T16:44:27.588] error: pyxis: tar: Skipping to next header
[2026-06-26T16:44:27.588] error: pyxis: tar: Exiting with failure status due to previous errors
[2026-06-26T16:44:27.588] error: pyxis: tar: This does not look like a tar archive
[2026-06-26T16:44:27.588] error: pyxis: tar: Skipping to next header
[2026-06-26T16:44:27.588] error: pyxis: tar: Exiting with failure status due to previous errors
[2026-06-26T16:44:27.588] error: pyxis: tar: This does not look like a tar archive
[2026-06-26T16:44:27.588] error: pyxis: tar: Skipping to next header
[2026-06-26T16:44:27.588] error: pyxis: tar: Exiting with failure status due to previous errors
[2026-06-26T16:44:27.588] error: pyxis: tar: This does not look like a tar archive
[2026-06-26T16:44:27.588] error: pyxis: tar: Skipping to next header
[2026-06-26T16:44:27.588] error: pyxis: tar: Archive contains ‘\016\321\311\031"rԗ\346\346\026V’ where numeric off_t value expected
[2026-06-26T16:44:27.588] error: pyxis: tar: Exiting with failure status due to previous errors
[2026-06-26T16:44:27.589] error: pyxis: couldn't start container
[2026-06-26T16:44:27.589] error: spank: required plugin spank_pyxis.so: task_init() failed with rc=-1
[2026-06-26T16:44:27.589] error: Failed to invoke spank plugin stack
[2026-06-26T16:44:27.649] error: pyxis: child 133747 failed with error code: 1
srun: error: x11-0436: task 0: Exited with exit code 1
salloc: Relinquishing job allocation 2837533
salloc: error: Unable to clean up job allocation 2837533: Socket timed out on send/recv operation
My devop's hint was that we compress our images with zstd which seems to not be supported with pyxis? I uncompressed and recompressed with gzip and it worked, so I guess he is right.
zstd is very fast I think it should be supported by pyxis
When trying to launch a node with one of our recent CCCL containers
cccl-devcontainers:cpp-llvm21-cuda99.8from gitlab I get the following error.My devop's hint was that we compress our images with zstd which seems to not be supported with pyxis? I uncompressed and recompressed with gzip and it worked, so I guess he is right.
zstd is very fast I think it should be supported by pyxis