Skip to content

Feat: decouple --group-affinity from the EP/PP/DP layout (placement-only affinity) - #1763

Merged
majieyue merged 1 commit into
intelligent-machine-learning:masterfrom
majieyue:feat/group-affinity-placement-only
Nov 16, 2026
Merged

majieyue merged 1 commit into
intelligent-machine-learning:masterfrom
majieyue:feat/group-affinity-placement-only

Conversation

@majieyue

Copy link
Copy Markdown
Collaborator

Motivation

With --node-group-strategy=ep_pp_dp configured together with
--group-affinity, the master fail-fasts at startup unless the declared
segment sizes exactly satisfy the validate_topology constraints (dense
DP divisible by G, EP groups fitting whole segment blocks, ...). Those
constraints couple a placement declaration with the communication
layout
— a coupling only meaningful when the layout is decided by the
declared groups themselves.

Failing job (work item 2026111600120657956): N=1152 pods, R=8 ranks/pod,
PP=12, EP=32, group-affinity of 9 segments x 128 nodes.
dense_dp = 1152*8/12 = 768 and 768 % 9 != 0, so the master dies with:

ValueError: node-group-strategy=ep_pp_dp requires dense_dp=768
(N*R/PP) to be divisible by G=9 segments.

Enumerating the valid space shows PP=12 + R=8 + 128-node groups can
never satisfy the constraints
: with G=9, PP must be a power of two
({2,4,8,16,32}); with PP=12, every group must be a multiple of 48 nodes.
The requested configuration is mathematically impossible — with
--enable-topology-rerank the declared sizes do not decide the layout
at all, so gating the startup on them is wrong.

Root cause

With rerank enabled, the layout is decided at join time by
GroupTopologySorter (observed psw groups become the segments, greedy
EP-PP column packing, no alignment constraint, crumbs degrade to a
warning). The declared sizes contribute nothing to it — the rerank
startup log already says so ("the declared affinities, if any, steer the
initial pod placement only"). But _init_group_affinity still ran
validate_topology and attached a NodeGroupSchedule on the
affinity path, making a purely declarative placement request abort the
master.

Changes

_init_group_affinity (master/node/dist_job_manager.py):

  • When --enable-topology-rerank is set, --group-affinity becomes
    placement-only: no NodeGroupSchedule is attached and
    validate_topology never runs. Any group sizes now combine with any
    PP/EP. Pods are labeled via the contiguous resolver, so the
    placement intent is unchanged and the EP/PP alignment comes from the
    observed psw.
  • --node-group-strategy is no longer required by --group-affinity:
    a placement-only affinity works standalone (with no strategy and no
    rerank it is purely a scheduling declaration, no parallel-size
    coupling).
  • The --expert-tensor-parallel-size ↔ --expert-model-parallel-size
    pairing check now only fires when the parallel sizes actually have a
    consumer (a strategy is set, or the rerank is on); a placement-only
    affinity skips it.
  • The backward-compatibility path is preserved: ep_pp_dp WITHOUT the
    topology rerank
    keeps the coupled behavior — validate_topology
    runs and the schedule is attached, because there the EP-PP column
    packing IS the layout. validate_topology, _resolve_stripe_group_id
    and the stripe resolver are untouched.
  • The placement-only mode is logged (group-affinity is placement-only with --enable-topology-rerank ...).

Also updated: --help texts for --group-affinity,
--node-group-strategy, --enable-topology-rerank and the parallel
sizes, plus the resolve_group_id docstring (the stripe mapping is now
documented as the legacy no-rerank path only).

Behavior matrix

Configuration Topology validation Pod labels World order
ep_pp_dp + rerank + affinity (the failing job) none (sum==replicas only) contiguous rerank, observed psw
ep_pp_dp + rerank, no affinity — none (free placement) rerank, observed psw
ep_pp_dp + affinity, no rerank (legacy) validate_topology EP-PP stripe original order
affinity alone (new) — contiguous original order

Tests

  • test_job_manager.py: 4 new tests — placement-only does not validate
    and attaches no schedule; affinity works standalone; the legacy
    no-rerank path still rejects a bad topology and stays unchanged; an
    end-to-end manager construction with rerank + affinity keeps no
    schedule. Existing validate_topology / stripe-mapping tests are
    untouched and green.
  • Neighbors all at baseline: test_node_group_strategy,
    test_topology_rerank, test_soft_group, test_pod_scaler.
  • Offline simulation of the failing job's parameters (N=1152, R=8,
    PP=12, EP=32, 9 psws x 128 nodes) against the real
    GroupTopologySorter: 0/288 EP slots straddle (deepep A2A never
    leaves its psw), 18/24 PP columns are single-segment with 6 mixed
    columns → 7 crossed p2p boundaries out of 264, and the dense-DP
    allreduce spans all 9 segments by design. A drifted topology
    (127/129-node segments) degrades to 1/288 EP straddles via the crumb
    packing.

Deployment note

Companion change: the dlrover-addon branch
feat/group-affinity-placement-only updates the --help wording of the
same flags (forwarding logic unchanged). The compatibility requirement
is one-directional: a master built from this change accepts any
combination; masters already in the fleet keep running legacy jobs
undisturbed.

With --enable-topology-rerank the declared --group-affinity becomes a
placement-only declaration: the master no longer validates the EP/PP/DP
parallel topology against the declared segment sizes and no longer
attaches a NodeGroupSchedule, so any group sizes now work with any PP/EP
(e.g. ep_pp_dp + 9 x 128 segments with PP=12). resolve_group_id keeps
the contiguous mapping for pod labels (scheduling/rack-group) and the
observed psw topology decides the rendezvous world order via the rerank.
--group-affinity also works standalone (no --node-group-strategy).

The legacy coupled behavior is preserved for ep_pp_dp WITHOUT rerank
(the EP-PP packing is then the layout): validate_topology still runs and
the schedule is still attached there; the EP-PP stripe resolver and
validate_topology are unchanged for that path.

Co-Authored-By: Claude <noreply@anthropic.com>
@codecov

codecov Bot commented Nov 16, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 82.28%. Comparing base (9676879) to head (c69a539).

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #1763      +/-   ##
==========================================
- Coverage   82.28%   82.28%   -0.01%     
==========================================
  Files         255      255              
  Lines       25432    25435       +3     
==========================================
+ Hits        20927    20929       +2     
- Misses       4505     4506       +1     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

@duck7216 duck7216 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@majieyue
majieyue merged commit 4cdbdb3 into intelligent-machine-learning:master Nov 16, 2026
12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants