Skip to content

Feat: save the checkpoint before relaunching an evicted pod - #1761

Merged
majieyue merged 1 commit into
intelligent-machine-learning:masterfrom
majieyue:feat/save-ckpt-on-eviction
Nov 11, 2026
Merged

majieyue merged 1 commit into
intelligent-machine-learning:masterfrom
majieyue:feat/save-ckpt-on-eviction

Conversation

@majieyue

Copy link
Copy Markdown
Collaborator

Background

When the cluster scheduler decides to evict a training pod (e.g. for resource defragmentation), the pod is force-killed and the job loses all training progress since the last periodic checkpoint. For large models the inter-checkpoint window can mean hours of lost GPU work.

This PR adds an opt-in "save checkpoint before eviction" flow: the master detects the eviction intent early (via a configurable pod label), asks the training to take an urgent checkpoint through the existing urgent-save protocol, and only then relaunches the evicted pod. The training side (atorch) saves synchronously and then sleeps, so the failover re-rendezvous resumes from the freshly saved checkpoint.

Changes

Eviction detection (bypasses the node state machine)

  • New NodeEventType.EVICTION. It is not a k8s event type and never goes through get_node_state_flow.
  • Node.is_evicted: set by _convert_pod_yaml_to_node when the pod carries the configured eviction label. Both the watch stream and the periodic list re-sync go through this conversion, so no extra LIST calls are added.
  • The label key/value are configured via the Context (eviction_pod_label_key / eviction_pod_label_value); empty key disables the detection. There is no built-in label — the job launcher wires its own.
  • _process_event converts any non-DELETED event of an evicted node into an EVICTION event and routes it to a dedicated branch.

Master-side orchestration (_handle_pod_eviction)

  • Gates: feature off / non-worker / non-RUNNING / job stopping or restarting / node no longer in the job nodes (stale pod from a previous generation) → skip with a log.
  • The urgent save is job-global: one in-flight save covers all pods evicted while it runs (dedup via _eviction_pending).
  • The master publishes the urgent-save command on the KV bus and a daemon thread polls the result — the node monitor thread is never blocked.
  • Wait ends on saved / error / timeout (eviction_save_timeout, default 900s); the evicted pods are relaunched in all three cases (on failure the job falls back to the last regular checkpoint).
  • After a completed save the training sleeps, so further evictions within one timeout window are relaunched directly without saving again.
  • Relaunch order per node: mark the old node not-relaunchable (so the DELETED event of the old pod cannot trigger a second relaunch) → _relaunch_node (the replacement pod is launched) → delete the old pod to complete the eviction.

Agent side: abort the in-flight save on failover (UrgentSaveListener.abort_in_flight)

  • The rank-0 listener now tracks the accepted-but-unconfirmed request.
  • When the agent stops the local workers (membership-change failover or a failure restart), the listener removes the marker — so a restarted training cannot consume a stale request — and publishes an error result, so the master stops waiting immediately instead of waiting for its deadline. The observer loop exits when its request is aborted or superseded.
  • No-op when no save is in flight, e.g. in the normal post-save failover of the eviction flow itself.

Tests

  • test_k8s_watcher.py: the eviction label flag is driven purely by the configured key/value.
  • test_job_manager.py::PodEvictionSaveTest (10 cases): event routing without touching the state machine, command publishing, dedup during an in-flight save, saved/error/timeout all relaunching, the skip-saving window after a completed save, non-worker/not-running/released/stale-node gates.
  • test_urgent_save.py: abort_in_flight (marker removed + error published), observer exit on abort and on supersede, in-flight cleared on save.

- new NodeEventType.EVICTION (bypasses the node state machine): the
  watcher flags nodes whose pod carries the configured eviction label
  (Context.eviction_pod_label_key/value, empty = detection disabled;
  the job launcher, e.g. dlrover-addon, wires the actual label), and
  _process_event routes them to a dedicated branch
- the master publishes an urgent-save request on the KV bus and a
  daemon thread waits for the result (Context.eviction_save_timeout,
  default 900s); once the save completes (or fails/times out) the
  evicted node is relaunched and the old pod deleted. One in-flight
  save covers all pods evicted while it runs; after a completed save,
  training sleeps so further evictions within one timeout window skip
  saving and relaunch directly. Pods no longer in the job nodes (stale
  pods from a previous generation) are skipped
- the rank-0 urgent-save listener now tracks the in-flight request and
  aborts it (marker removal + error result) when the agent stops the
  local workers for a failover/restart, so the master stops waiting
  immediately instead of waiting for its deadline

Co-Authored-By: Claude <nor***@anthropic.com>
@codecov

codecov Bot commented Nov 11, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 90.00000% with 15 lines in your changes missing coverage. Please review.
✅ Project coverage is 82.24%. Comparing base (a80b301) to head (d2e773b).

Files with missing lines Patch % Lines
dlrover/python/master/node/dist_job_manager.py 91.58% 9 Missing ⚠️
dlrover/python/elastic_agent/torch/training.py 14.28% 6 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master    #1761      +/-   ##
==========================================
- Coverage   82.26%   82.24%   -0.02%     
==========================================
  Files         255      255              
  Lines       25282    25432     +150     
==========================================
+ Hits        20797    20917     +120     
- Misses       4485     4515      +30     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

@majieyue
majieyue merged commit 9676879 into intelligent-machine-learning:master Nov 11, 2026
11 of 12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant