Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
43 changes: 43 additions & 0 deletions swe-paddle/tasks/PaddlePaddle__Paddle-36684/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,43 @@
# PaddlePaddle__Paddle-36684

This directory converts Paddle PR #36684 into a SWE-Paddle community task candidate.

## Source

| Field | Value |
| --- | --- |
| Repo | `PaddlePaddle/Paddle` |
| PR | [36684](https://github.com/PaddlePaddle/Paddle/pull/36684) |
| PR title | `fleet support elastic scale up/down` |
| Base commit | `9a9345fa4dc77be655811d8e484b99cb9ff5f356` |
| Merged at | `2021-11-11T06:27:42Z` |
| Task type | `feature_enhancement` |
| Resource | CPU |

## Summary

Enable Fleet elastic jobs to use a node-count range and keep ranks, hosts, and endpoints consistent across scale-out and scale-in transitions.

## Why This Is A Good SWE-Paddle Candidate

- It covers a real distributed-launch workflow with clear behavior before and after the change.
- The production change spans range parsing, readiness decisions, host membership, and endpoint updates rather than a single local condition.
- The candidate executes assertions and scenarios from the source PR tests without requiring a live etcd service or GPU.

## Files

- `proposal.md`: candidate proposal for maintainer triage.
- `instruction.md`: self-contained problem statement for the coding agent.
- `solution/code.patch`: gold patch from the merged PR.
- `tests/test.patch`: source-PR tests plus the controlled source adapter.
- `tests/test.sh`: minimal target test command.
- `environment/README.md`: environment notes for reproduction.
- `README.md`: task overview and verification entrypoint.

## Verification

```bash
bash tests/test.sh
```

Expected behavior: applying `tests/test.patch` to `base_commit` should fail on the target behavior; applying both `tests/test.patch` and `solution/code.patch` should pass the target tests.
27 changes: 27 additions & 0 deletions swe-paddle/tasks/PaddlePaddle__Paddle-36684/environment/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
# Environment Notes

This candidate is part of the SWE-Paddle community task set.

## Expected Environment

- Repository: `PaddlePaddle/Paddle`
- Base commit: `9a9345fa4dc77be655811d8e484b99cb9ff5f356`
- Resource: CPU
- GPU required: no
- Build path: no source build is required; a controlled AST overlay executes the checkout implementation with the source PR's unittest methods and local doubles for etcd and device discovery.

## Run Order

1. Check out `PaddlePaddle/Paddle` at the base commit.
2. Apply `tests/test.patch`.
3. Run `bash tests/test.sh`; the target behavior should fail before the fix.
4. Apply `solution/code.patch`.
5. Run `bash tests/test.sh` again; the target behavior should pass after the gold patch.

## Minimal Test Command

```bash
bash tests/test.sh
```

The verifier is responsible for deriving stable F2P and P2P node IDs from repeated runs.
19 changes: 19 additions & 0 deletions swe-paddle/tasks/PaddlePaddle__Paddle-36684/instruction.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
# 支持 Fleet 弹性训练按节点范围扩缩容

## 详细描述

目前 Fleet 弹性训练只能按固定节点数启动。实际运行中,可用机器数量可能会随资源调度发生变化,用户即使配置了类似 `2:4` 的节点范围,也无法让任务在 2 到 4 个节点之间正常等待、启动和扩缩容。

需要支持用最小值和最大值描述可接受的节点数量。达到最大节点数时可以立即开始;节点数位于范围内时可以等待更多节点,并在等待超时后使用当前节点继续运行;低于最小节点数时继续等待。运行期间节点加入或退出后,训练进程应拿到正确的主机和 endpoint 信息。

## 验收说明

- 支持固定节点数和 `MIN:MAX` 两种配置方式,并正确判断当前节点是否满足启动条件。
- 节点扩容或缩容后,训练使用的主机列表、rank 和 endpoint 信息应与当前节点一致。
- 原有固定节点数下的故障恢复和 launch 流程保持不变。

## 技术要求

- 熟悉 Python。
- 熟悉 PaddlePaddle Fleet 分布式启动流程。
- 了解弹性训练中的节点发现和进程环境变量。
55 changes: 55 additions & 0 deletions swe-paddle/tasks/PaddlePaddle__Paddle-36684/proposal.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,55 @@
# Task Proposal: PaddlePaddle__Paddle-36684

## 1. 来源信息

- Instance ID:`PaddlePaddle__Paddle-36684`
- PR 链接:https://github.com/PaddlePaddle/Paddle/pull/36684
- PR 标题:`fleet support elastic scale up/down`
- `base_commit`:`9a9345fa4dc77be655811d8e484b99cb9ff5f356`
- merged 时间:`2021-11-11T06:27:42Z`
- 你的身份:熟悉该模块的 contributor
- 后续联系人:TBD

## 2. 问题一句话

Fleet 弹性训练需要接受节点数量范围,并在节点加入或退出后更新训练所需的主机、rank 和 endpoint 信息。

## 3. 为什么适合作为 SWE-Paddle 样本

- **真实性**:来源于 Fleet 弹性训练对动态资源调度的实际支持需求。
- **代表性**:覆盖节点范围解析、启动条件判断和扩缩容后的环境更新。
- **边界清楚**:production change 集中在 Python 分布式 elastic launch 流程。
- **非平凡性**:需要同时处理固定节点模式、范围模式、超时等待以及扩缩容后的 rank/endpoint 连续性。
- **环境友好性**:测试使用来源 PR 的 unittest 场景和 controlled doubles,不需要启动真实 etcd、训练进程或 GPU。

## 4. 任务类型和标签

- 任务类型:`feature_enhancement`
- 执行后端:`cpu`
- 设备范围:`cpu_only`
- 模块标签:`[distributed, fleet, elastic, launch]`

## 5. 验证思路

- 目标测试命令:`bash tests/test.sh`
- 目标测试文件:`test/swe_paddle/test_pr36684_elastic_scale.py`
- 修复前预期:固定节点的 elastic enablement 继续通过;来源 PR 中范围匹配和扩缩容用例因 Base 不支持范围节点数而失败。
- 修复后预期:固定节点 P2P 与两个范围扩缩容 F2P 全部通过。
- P2P 候选:来源 PR `TestElasticInit.test_enable_elastic` 对既有 elastic enablement 的验证。

## 6. 环境与资源

- 资源需求:CPU
- Paddle 来源:`PaddlePaddle/Paddle` source checkout at `base_commit`
- 是否能提供 Docker:暂无
- patch 类型:Python-only
- 环境建议:使用 AST overlay 执行 checkout 中的真实控制流,并直接调用来源 PR 的 unittest 方法;etcd 和设备探测使用 controlled doubles。
- 最小测试命令:`bash tests/test.sh`
- 是否有 oracle 日志:由 SWE-Paddle verifier 结果另行维护

## 7. 风险自查

- 泄露风险:instruction 只描述外部行为,没有给出 Gold patch 的具体修改方式。
- 环境风险:无需真实 etcd、分布式集群、网络服务或 GPU。
- flaky 风险:不依赖真实节点变化、计时竞态或后台训练进程。
- 拆分风险:测试只覆盖节点范围匹配及扩缩容环境更新这一组相互依赖的行为。
Loading