Repository navigation
Feat: decouple --group-affinity from the EP/PP/DP layout (placement-only affinity) - #1763
Merged
majieyue merged 1 commit intoNov 16, 2026
Conversation
With --enable-topology-rerank the declared --group-affinity becomes a placement-only declaration: the master no longer validates the EP/PP/DP parallel topology against the declared segment sizes and no longer attaches a NodeGroupSchedule, so any group sizes now work with any PP/EP (e.g. ep_pp_dp + 9 x 128 segments with PP=12). resolve_group_id keeps the contiguous mapping for pod labels (scheduling/rack-group) and the observed psw topology decides the rendezvous world order via the rerank. --group-affinity also works standalone (no --node-group-strategy). The legacy coupled behavior is preserved for ep_pp_dp WITHOUT rerank (the EP-PP packing is then the layout): validate_topology still runs and the schedule is still attached there; the EP-PP stripe resolver and validate_topology are unchanged for that path. Co-Authored-By: Claude <noreply@anthropic.com>
majieyue
requested review from
BalaBalaYi,
bikemechanic,
duck7216,
samplise,
way-zer and
workingloong
as code owners
November 16, 2026 07:20
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #1763 +/- ##
==========================================
- Coverage 82.28% 82.28% -0.01%
==========================================
Files 255 255
Lines 25432 25435 +3
==========================================
+ Hits 20927 20929 +2
- Misses 4505 4506 +1 ☔ View full report in Codecov by Harness. |
majieyue
merged commit Nov 16, 2026
4cdbdb3
into
intelligent-machine-learning:master
12 checks passed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
With
--node-group-strategy=ep_pp_dpconfigured together with--group-affinity, the master fail-fasts at startup unless the declaredsegment sizes exactly satisfy the
validate_topologyconstraints (denseDP divisible by G, EP groups fitting whole segment blocks, ...). Those
constraints couple a placement declaration with the communication
layout — a coupling only meaningful when the layout is decided by the
declared groups themselves.
Failing job (work item 2026111600120657956): N=1152 pods, R=8 ranks/pod,
PP=12, EP=32, group-affinity of 9 segments x 128 nodes.
dense_dp = 1152*8/12 = 768and768 % 9 != 0, so the master dies with:Enumerating the valid space shows PP=12 + R=8 + 128-node groups can
never satisfy the constraints: with G=9, PP must be a power of two
({2,4,8,16,32}); with PP=12, every group must be a multiple of 48 nodes.
The requested configuration is mathematically impossible — with
--enable-topology-rerankthe declared sizes do not decide the layoutat all, so gating the startup on them is wrong.
Root cause
With rerank enabled, the layout is decided at join time by
GroupTopologySorter(observed psw groups become the segments, greedyEP-PP column packing, no alignment constraint, crumbs degrade to a
warning). The declared sizes contribute nothing to it — the rerank
startup log already says so ("the declared affinities, if any, steer the
initial pod placement only"). But
_init_group_affinitystill ranvalidate_topologyand attached aNodeGroupScheduleon theaffinity path, making a purely declarative placement request abort the
master.
Changes
_init_group_affinity(master/node/dist_job_manager.py):--enable-topology-rerankis set,--group-affinitybecomesplacement-only: no
NodeGroupScheduleis attached andvalidate_topologynever runs. Any group sizes now combine with anyPP/EP. Pods are labeled via the contiguous resolver, so the
placement intent is unchanged and the EP/PP alignment comes from the
observed psw.
--node-group-strategyis no longer required by--group-affinity:a placement-only affinity works standalone (with no strategy and no
rerank it is purely a scheduling declaration, no parallel-size
coupling).
--expert-tensor-parallel-size↔--expert-model-parallel-sizepairing check now only fires when the parallel sizes actually have a
consumer (a strategy is set, or the rerank is on); a placement-only
affinity skips it.
topology rerank keeps the coupled behavior —
validate_topologyruns and the schedule is attached, because there the EP-PP column
packing IS the layout.
validate_topology,_resolve_stripe_group_idand the stripe resolver are untouched.
group-affinity is placement-only with --enable-topology-rerank ...).Also updated:
--helptexts for--group-affinity,--node-group-strategy,--enable-topology-rerankand the parallelsizes, plus the
resolve_group_iddocstring (the stripe mapping is nowdocumented as the legacy no-rerank path only).
Behavior matrix
validate_topologyTests
test_job_manager.py: 4 new tests — placement-only does not validateand attaches no schedule; affinity works standalone; the legacy
no-rerank path still rejects a bad topology and stays unchanged; an
end-to-end manager construction with rerank + affinity keeps no
schedule. Existing
validate_topology/ stripe-mapping tests areuntouched and green.
test_node_group_strategy,test_topology_rerank,test_soft_group,test_pod_scaler.PP=12, EP=32, 9 psws x 128 nodes) against the real
GroupTopologySorter: 0/288 EP slots straddle (deepep A2A neverleaves its psw), 18/24 PP columns are single-segment with 6 mixed
columns → 7 crossed p2p boundaries out of 264, and the dense-DP
allreduce spans all 9 segments by design. A drifted topology
(127/129-node segments) degrades to 1/288 EP straddles via the crumb
packing.
Deployment note
Companion change: the
dlrover-addonbranchfeat/group-affinity-placement-onlyupdates the--helpwording of thesame flags (forwarding logic unchanged). The compatibility requirement
is one-directional: a master built from this change accepts any
combination; masters already in the fleet keep running legacy jobs
undisturbed.