Skip to content

WIP: In place restart demo - #1762

Open
samplise wants to merge 6 commits into
intelligent-machine-learning:masterfrom
samplise:healthy-recovery
Open

samplise wants to merge 6 commits into
intelligent-machine-learning:masterfrom
samplise:healthy-recovery

Conversation

@samplise

Copy link
Copy Markdown
Collaborator

What changes were proposed in this pull request?

Added an experimental four-GPU MoE in-place recovery example under examples/in_place_restart/:

  • Deterministic replicated-expert MoE training with PyTorch DDP.
  • Atomic checkpoint save/restore and data-cursor recovery.
  • nccl2 process-group reconfiguration using PyTorch’s experimental reconfigure APIs.
  • A local supervisor that restarts only the failed rank while preserving healthy processes.
  • Explicit PID and recovery-status logging to verify healthy-rank continuity.
  • Image preflight checks, a Dockerfile, launch scripts, and documentation.

Why are the changes needed?

Current recovery normally restarts all ranks and repeats model, optimizer, and distributed-runtime initialization. This example validates a lower-overhead approach where only the failed rank gets a new process, while healthy ranks keep their PIDs, CUDA contexts, model parameters, and optimizer objects and rejoin a reconfigured process group.

Does this PR introduce any user-facing change?

Yes, but only as a new opt-in example. It introduces no changes to existing training behavior. The example requires PyTorch 2.14+, the experimental nccl2 backend, and NCCL 2.28+.

How was this patch tested?

Passed Python compilation, style, shell syntax, and recovery-protocol smoke checks.
Passed Docker BuildKit static validation.
Validated with a four-GPU run:- Rank 2 failed at step 23.

  • Rank 2 was replaced: PID 860 → 1208.
  • Ranks 0, 1, and 3 retained PIDs 858, 859, and 861.
  • All ranks restored checkpoint step 20 and resumed successfully.
  • All ranks reached step 40 with recoveries=1.

@codecov

codecov Bot commented Nov 11, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 82.29%. Comparing base (a80b301) to head (aeb77b7).
⚠️ Report is 1 commits behind head on master.

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #1762      +/-   ##
==========================================
+ Coverage   82.26%   82.29%   +0.03%     
==========================================
  Files         255      255              
  Lines       25282    25432     +150     
==========================================
+ Hits        20797    20929     +132     
- Misses       4485     4503      +18     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant