ci: build CI image against nvidia-resiliency-ext main

[hexinw-nvidia]

What

Builds the CI image against upstream nvidia-resiliency-ext main instead of the locked 0.6.0 PyPI wheel, so the full CI suite runs against the latest NVRx.

Experiment only — not intended to merge.

How

A single uv pip install layer appended to the main stage of docker/Dockerfile.ci.dev.

This is deliberately not done via pyproject.toml + uv.lock. Every uv sync in this repo passes --locked (Dockerfile.ci.dev:65, Dockerfile.ci.lts:51, docker/common/install.sh:141, and cicd-main.yml:355 in the linting job), which rejects a pyproject.toml change without a matching relock. An imperative install sidesteps all four gates.

Overridable via --build-arg NVRX_REF=<sha|branch|tag>; defaults to main.

Why install with dependencies

NVRx main adds a substantial set of runtime dependencies over the pinned 0.6.0:

langchain-core, langchain-openai, mcp, logsage, setproctitle, uvicorn[standard], pydantic-settings, slowapi, slack-bolt, slack-sdk, fastapi

--no-deps would install NVRx main and leave it un-importable, so the layer installs with dependencies and guards the two known hazards:

• torch — reuses the override-dependencies trick from pyproject.toml:194-198 (torch; sys_platform == 'never', plus torchvision/triton) so NVRx's torch>=2.3.0 cannot pull a PyPI CPU wheel over the base image's CUDA build.
• pydantic — constrained to <2.14. Per the note at pyproject.toml:209-212, pydantic 2.14 breaks langchain_core's module-level RunnablePassthrough() instantiation. That was previously a transitive concern; NVRx main depends on langchain-core directly, making it an import-time failure.

The layer ends with uv pip freeze | grep -i resiliency and an import smoke test. Since main floats, the container build log is the only record of which NVRx commit was actually tested.

Test plan

Labelled Run functional tests (scope=mr-github, n_repeat=5, golden-value comparison) rather than Run tests. NVRx owns the async-checkpoint and in-process-restart paths, and this jumps a dozen dependency versions — the lightweight 4-step lane would not surface a numerics or checkpointing regression.

Existing gpt3_mcore_te_tp2_pp1_gdn_no_nvrx_{sync,async,async_mcore} cases uninstall NVRx in their BEFORE_SCRIPT and are unaffected.

🤖 Generated with Claude Code

[copy-pr-bot]

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

[hexinw-nvidia]

/ok to test 64a6621