Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
44 changes: 44 additions & 0 deletions swe-paddle/tasks/PaddlePaddle__Paddle-79633/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,44 @@
# PaddlePaddle__Paddle-79633

This directory converts Paddle PR #79633 into a SWE-Paddle community task candidate.

## Source

| Field | Value |
| --- | --- |
| Repo | `PaddlePaddle/Paddle` |
| PR | [79633](https://github.com/PaddlePaddle/Paddle/pull/79633) |
| PR title | `[Distributed Strategy] Fix KV server hangs under concurrent requests` |
| Base commit | `58354a509a8d60b2cb3cdf6ead63a6c845eefd23` |
| Merged at | `2026-08-10T12:30:53Z` |
| Task type | `bug_fix` |
| Resource | CPU |

## Summary

Fix the distributed launch KV server so concurrent registration and incomplete requests do not block other nodes from completing startup synchronization.

## Why This Is A Good SWE-Paddle Candidate

- The failure is observable as distributed launch requests hanging during node registration and synchronization.
- The upstream PR includes focused tests for concurrent requests, stalled connections, and clean shutdown.
- The production change is limited to one Python file and can be verified on CPU using loopback networking.
- The task does not require a GPU, external service, dataset, or model checkpoint.

## Files

- `proposal.md`: candidate proposal for maintainer triage.
- `instruction.md`: self-contained problem statement for the coding agent.
- `solution/code.patch`: gold patch from the merged PR.
- `tests/test.patch`: exact upstream test patch exposing the target behavior.
- `tests/test.sh`: minimal target test command.
- `environment/README.md`: environment notes for reproduction.
- `README.md`: task overview and verification entrypoint.

## Verification

```bash
bash tests/test.sh
```

Expected behavior: applying `tests/test.patch` to `base_commit` should fail on the target behavior; applying both `tests/test.patch` and `solution/code.patch` should pass the target tests.
27 changes: 27 additions & 0 deletions swe-paddle/tasks/PaddlePaddle__Paddle-79633/environment/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
# Environment Notes

This candidate is part of the SWE-Paddle community task set.

## Expected Environment

- Repository: `PaddlePaddle/Paddle`
- Base commit: `58354a509a8d60b2cb3cdf6ead63a6c845eefd23`
- Resource: CPU
- GPU required: no
- Build path: Python-only source change; use a compatible installed Paddle runtime or source build with `pytest` and `httpx` available.

## Run Order

1. Check out `PaddlePaddle/Paddle` at the base commit.
2. Apply `tests/test.patch`.
3. Run `bash tests/test.sh`; the target behavior should fail before the fix.
4. Apply `solution/code.patch`.
5. Run `bash tests/test.sh` again; the target behavior should pass after the gold patch.

## Minimal Test Command

```bash
bash tests/test.sh
```

The verifier is responsible for deriving stable F2P and P2P node IDs from repeated runs.
21 changes: 21 additions & 0 deletions swe-paddle/tasks/PaddlePaddle__Paddle-79633/instruction.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
# 修复大规模分布式启动时 KV server 请求阻塞

## 详细描述

Paddle 的分布式启动流程会通过 KV server 收集各节点信息。节点数量较多、多个节点同时注册时,部分 `put` 或 `get_prefix` 请求可能长时间没有响应,导致所有节点一直等待,训练任务无法正常启动。

此外,如果某个客户端建立连接后没有把请求发送完整,这个连接可能一直占用服务端处理能力,后续正常请求也会受到影响。

需要让 KV server 在并发请求和未完成请求出现时仍能继续提供服务,并确保停止服务时不会卡住或遗留监听端口。

## 验收说明

- 多个客户端同时写入并读取节点信息时,请求能够完成,记录不能丢失或相互覆盖。
- 客户端连接后没有完成请求时,服务端应在合理时间内释放该连接,其他正常请求仍能继续处理。
- KV server 停止时不能卡住,并应释放监听端口;已有的 `put`、`get` 和 `get_prefix` 行为保持不变。

## 技术要求

- 熟悉 Python HTTP server 和并发请求处理。
- 熟悉 PaddlePaddle 分布式 launch 流程。
- 熟悉网络服务的启动、停止和异常连接处理。
56 changes: 56 additions & 0 deletions swe-paddle/tasks/PaddlePaddle__Paddle-79633/proposal.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,56 @@
# Task Proposal: PaddlePaddle__Paddle-79633

## 1. 来源信息

- Instance ID:`PaddlePaddle__Paddle-79633`
- PR 链接:https://github.com/PaddlePaddle/Paddle/pull/79633
- PR 标题:`[Distributed Strategy] Fix KV server hangs under concurrent requests`
- `base_commit`:`58354a509a8d60b2cb3cdf6ead63a6c845eefd23`
- merged 时间:`2026-08-10T12:30:53Z`
- 你的身份:熟悉该模块的 contributor
- 后续联系人:TBD

## 2. 问题一句话

分布式启动使用的 KV server 在并发注册或遇到未完成请求时可能阻塞,导致其他节点无法继续完成启动同步。

## 3. 为什么适合作为 SWE-Paddle 样本

- **真实性**:问题来自大规模分布式启动过程中节点注册和信息同步的真实阻塞场景。
- **代表性**:覆盖 Python 网络服务的并发处理、异常连接回收和服务生命周期管理。
- **边界清楚**:production change 仅涉及 KV server,测试也只访问本机临时端口。
- **非平凡性**:修复需要同时保证并发请求、半开连接和正常停止,不是简单修改返回值或错误信息。
- **环境友好性**:原 PR 测试使用 CPU、loopback 网络和系统分配的临时端口,不需要 GPU、外部服务或数据集。

## 4. 任务类型和标签

- 任务类型:`bug_fix`
- 执行后端:`cpu`
- 设备范围:`cpu_only`
- 模块标签:`[distributed_launch, kv_server, concurrency, networking]`

## 5. 验证思路

- 目标测试命令:`bash tests/test.sh`
- 目标测试文件:`test/legacy_test/test_kv_server.py`
- 修复前预期:服务停止测试通过;并发请求测试和未完成请求超时测试失败。
- 修复后预期:三个原 PR 测试全部通过,服务可以并发响应、释放异常连接并正常停止。
- P2P 候选:`TestKVServerStop::test_stop_is_clean_and_idempotent_state`
- F2P 候选:`TestKVServerConcurrent::test_concurrent_put_get_prefix`、`TestKVServerRequestTimeout::test_half_open_connection_is_released_after_timeout`

## 6. 环境与资源

- 资源需求:CPU
- Paddle 来源:`PaddlePaddle/Paddle` source checkout at `base_commit`
- 是否能提供 Docker:暂无
- patch 类型:Python-only
- 环境建议:使用与源码兼容的 Paddle Python 环境;测试仅使用本机 loopback 网络和临时端口。
- 最小测试命令:`bash tests/test.sh`
- 是否有 oracle 日志:由 SWE-Paddle verifier 结果另行维护

## 7. 风险自查

- 泄露风险:instruction 只描述可观察的阻塞、恢复和停止行为,没有给出 Gold patch 的具体实现。
- 环境风险:测试依赖 `httpx`,该依赖随 Paddle launch 环境提供;不需要外部网络。
- flaky 风险:测试使用系统分配的临时端口,并设置有界等待;不依赖真实集群竞态。
- 拆分风险:PR 只解决 KV server 在并发和异常连接下阻塞这一项问题,三个测试共同验证同一服务可用性契约。
46 changes: 46 additions & 0 deletions swe-paddle/tasks/PaddlePaddle__Paddle-79633/solution/code.patch
Original file line number Diff line number Diff line change
@@ -0,0 +1,46 @@
diff --git a/python/paddle/distributed/launch/utils/kv_server.py b/python/paddle/distributed/launch/utils/kv_server.py
index 279d03eba00074a8af08888fdae89ebce33a573b..ef448983764b3fddf058cbf7c695316e7738de8d 100644
--- a/python/paddle/distributed/launch/utils/kv_server.py
+++ b/python/paddle/distributed/launch/utils/kv_server.py
@@ -15,13 +15,20 @@
import http.server as SimpleHTTPServer
import json
import threading
-from http.server import HTTPServer
+from http.server import ThreadingHTTPServer
from multiprocessing import Process

from .topology import SingleNodeTopology


class KVHandler(SimpleHTTPServer.SimpleHTTPRequestHandler):
+ # StreamRequestHandler.setup() applies this as a socket timeout, and
+ # BaseHTTPRequestHandler.handle_one_request() turns the resulting TimeoutError into a
+ # connection close. Without it a peer that connects but never finishes sending its
+ # request line blocks the handler in rfile.readline() forever; on a single-threaded
+ # server that wedges the whole KV store and no other node can ever register.
+ timeout = 30
+
def do_GET(self):
with self.server.kv_lock:
ret = {}
@@ -67,7 +74,18 @@ class KVHandler(SimpleHTTPServer.SimpleHTTPRequestHandler):
return


-class KVServer(HTTPServer):
+class KVServer(ThreadingHTTPServer):
+ # The default socketserver.TCPServer.request_queue_size is 5, i.e. listen(5).
+ # At 768 nodes every launcher polls put()/get_prefix() every 0.5s over HTTP/1.0
+ # (one new TCP connection per request), which overflows a 5-deep accept queue.
+ # Overflowed SYNs are dropped, and because KVClient uses timeout=None the client
+ # then waits out the ~127s kernel SYN-retransmit timeout, so a few nodes can never
+ # register and sync_peers livelocks. Serve requests concurrently and give the
+ # accept queue enough room. kv is already guarded by kv_lock, so KVHandler is
+ # safe to run on multiple threads.
+ request_queue_size = 2048
+ daemon_threads = True
+
def __init__(self, port):
super().__init__(('', port), KVHandler)
self.kv_lock = threading.Lock()
Loading