[fix] Guarantee even node distribution for simple storage units - #144
Conversation
Placement group SPREAD scheduling is best-effort and can pack multiple storage units onto the same node; switch to explicit round-robin NodeAffinitySchedulingStrategy so units are evenly split across all alive Ray nodes regardless of how num_data_storage_units compares to the node count. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
CLA Signature Guide@OutstanderWang , thanks for your pull request. The following commit(s) are not associated with a signed Contributor License Agreement (CLA).
To sign CLA, click here. To check if your email is configured correctly, refer to the FAQs. Once you've signed the CLA or updating your email, please comment |
|
/check-cla |
CLA Signature Guide@OutstanderWang , thanks for your pull request. The following commit(s) are not associated with a signed Contributor License Agreement (CLA).
To sign CLA, click here. To check if your email is configured correctly, refer to the FAQs. Once you've signed the CLA or updating your email, please comment |
3b67867 to
9f0212f
Compare
CLA Signature PassOutstanderWang, thanks for your pull request. All authors of the commits have signed the CLA. 👍 |
| total_storage_size: null | ||
| # Number of distributed storage units. | ||
| # Recommended: >= 2 x number of nodes for load balancing. | ||
| # Number of distributed storage units. Units are round-robin scheduled across all |
There was a problem hiding this comment.
Please also update the config in https://github.com/Ascend/TransferQueue/blob/main/scripts/performance_test/perftest_config.yaml
CLA Signature Guide@OutstanderWang , thanks for your pull request. The following commit(s) are not associated with a signed Contributor License Agreement (CLA).
To sign CLA, click here. To check if your email is configured correctly, refer to the FAQs. Once you've signed the CLA or updating your email, please comment |
Mirror the num_data_storage_units documentation from transfer_queue/config.yaml so the perf-test config reflects the round-robin node scheduling behavior. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
0eb642c to
240c9e2
Compare
CLA Signature Guide@OutstanderWang , thanks for your pull request. The following commit(s) are not associated with a signed Contributor License Agreement (CLA).
To sign CLA, click here. To check if your email is configured correctly, refer to the FAQs. Once you've signed the CLA or updating your email, please comment |
3 participants
Motivation
SimpleStoragefans experience data out acrossnum_data_storage_unitsRay actors. For memory and network bandwidth to be balanced across the cluster, those units must be spread evenly over the available Ray nodes — otherwise a subset of nodes absorbs a disproportionate share of the storage load and becomes a hotspot. The current placement relies on a placement group with theSPREADstrategy, which does not actually guarantee this.What was wrong
SPREADis best-effort, not a guarantee.initialize_simple_storagecreated a placement group viaget_placement_group(num_data_storage_units), which builds one CPU bundle per unit withray.util.placement_group(..., strategy="SPREAD"). Ray'sSPREADis explicitly documented as best-effort: when scheduling is constrained (resource pressure, timing, node availability) it silently falls back to packing multiple bundles — and therefore multiple storage units — onto the same node.SPREADgives no even-split guarantee regardless of hownum_data_storage_unitscompares to the node count, and its strict siblingSTRICT_SPREADisn't a usable alternative here because it fails outright oncenum_actors > num_nodes(the common case, since the recommended setting is ≥ 2× nodes). The result: uneven memory/bandwidth distribution and node hotspots that the "spread" was supposed to prevent.What this changes
NodeAffinitySchedulingStrategy. New helperget_node_round_robin_scheduling_strategies(num_actors)intransfer_queue/utils/common.pyenumerates all currently alive Ray nodes (ray.nodes()filtered byAlive, sorted byNodeIDfor deterministic ordering) and returns one strategy per actor, assigning actoritoalive_node_ids[i % len(alive_node_ids)]withsoft=False(hard affinity). This guarantees each node receivesfloor(num_actors / num_nodes)orceil(num_actors / num_nodes)units — an even split by construction, for any ratio of units to nodes.initialize_simple_storagenow schedules per-unit. It callsget_node_round_robin_scheduling_strategies(num_data_storage_units)and passesscheduling_strategy=strategies[rank]to eachSimpleStorageUnit.options(...), replacing the previousplacement_group+placement_group_bundle_indexwiring. The creation log line now records the targetnode_idfor each unit, making the distribution observable.RuntimeError("No alive Ray nodes found. Is Ray initialized?")instead of proceeding with an undefined placement.config.yamldocuments that units are round-robin scheduled across all alive nodes for an even per-node split, and keeps the "≥ 2× nodes" recommendation (now framed as "so each node hosts multiple units").Tests
Verified on a Ray cluster that
num_data_storage_unitsunits are distributed evenly across alive nodes for bothunits ≤ nodesandunits > nodescases, with each unit's target node confirmed via the new creation log line. ExistingSimpleStorageunit and e2e lifecycle tests continue to pass.