Repository navigation
Conversation
- new NodeEventType.EVICTION (bypasses the node state machine): the watcher flags nodes whose pod carries the configured eviction label (Context.eviction_pod_label_key/value, empty = detection disabled; the job launcher, e.g. dlrover-addon, wires the actual label), and _process_event routes them to a dedicated branch - the master publishes an urgent-save request on the KV bus and a daemon thread waits for the result (Context.eviction_save_timeout, default 900s); once the save completes (or fails/times out) the evicted node is relaunched and the old pod deleted. One in-flight save covers all pods evicted while it runs; after a completed save, training sleeps so further evictions within one timeout window skip saving and relaunch directly. Pods no longer in the job nodes (stale pods from a previous generation) are skipped - the rank-0 urgent-save listener now tracks the in-flight request and aborts it (marker removal + error result) when the agent stops the local workers for a failover/restart, so the master stops waiting immediately instead of waiting for its deadline Co-Authored-By: Claude <nor***@anthropic.com>
majieyue
requested review from
BalaBalaYi,
bikemechanic,
duck7216,
samplise,
way-zer and
workingloong
as code owners
November 11, 2026 06:32
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #1761 +/- ##
==========================================
- Coverage 82.26% 82.24% -0.02%
==========================================
Files 255 255
Lines 25282 25432 +150
==========================================
+ Hits 20797 20917 +120
- Misses 4485 4515 +30 ☔ View full report in Codecov by Harness. |
majieyue
merged commit Nov 11, 2026
9676879
into
intelligent-machine-learning:master
11 of 12 checks passed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Background
When the cluster scheduler decides to evict a training pod (e.g. for resource defragmentation), the pod is force-killed and the job loses all training progress since the last periodic checkpoint. For large models the inter-checkpoint window can mean hours of lost GPU work.
This PR adds an opt-in "save checkpoint before eviction" flow: the master detects the eviction intent early (via a configurable pod label), asks the training to take an urgent checkpoint through the existing urgent-save protocol, and only then relaunches the evicted pod. The training side (atorch) saves synchronously and then sleeps, so the failover re-rendezvous resumes from the freshly saved checkpoint.
Changes
Eviction detection (bypasses the node state machine)
Master-side orchestration (_handle_pod_eviction)
Agent side: abort the in-flight save on failover (UrgentSaveListener.abort_in_flight)
Tests