diff --git a/swe-paddle/tasks/PaddlePaddle__Paddle-79633/README.md b/swe-paddle/tasks/PaddlePaddle__Paddle-79633/README.md new file mode 100644 index 000000000..84650ab27 --- /dev/null +++ b/swe-paddle/tasks/PaddlePaddle__Paddle-79633/README.md @@ -0,0 +1,44 @@ +# PaddlePaddle__Paddle-79633 + +This directory converts Paddle PR #79633 into a SWE-Paddle community task candidate. + +## Source + +| Field | Value | +| --- | --- | +| Repo | `PaddlePaddle/Paddle` | +| PR | [79633](https://github.com/PaddlePaddle/Paddle/pull/79633) | +| PR title | `[Distributed Strategy] Fix KV server hangs under concurrent requests` | +| Base commit | `58354a509a8d60b2cb3cdf6ead63a6c845eefd23` | +| Merged at | `2026-08-10T12:30:53Z` | +| Task type | `bug_fix` | +| Resource | CPU | + +## Summary + +Fix the distributed launch KV server so concurrent registration and incomplete requests do not block other nodes from completing startup synchronization. + +## Why This Is A Good SWE-Paddle Candidate + +- The failure is observable as distributed launch requests hanging during node registration and synchronization. +- The upstream PR includes focused tests for concurrent requests, stalled connections, and clean shutdown. +- The production change is limited to one Python file and can be verified on CPU using loopback networking. +- The task does not require a GPU, external service, dataset, or model checkpoint. + +## Files + +- `proposal.md`: candidate proposal for maintainer triage. +- `instruction.md`: self-contained problem statement for the coding agent. +- `solution/code.patch`: gold patch from the merged PR. +- `tests/test.patch`: exact upstream test patch exposing the target behavior. +- `tests/test.sh`: minimal target test command. +- `environment/README.md`: environment notes for reproduction. +- `README.md`: task overview and verification entrypoint. + +## Verification + +```bash +bash tests/test.sh +``` + +Expected behavior: applying `tests/test.patch` to `base_commit` should fail on the target behavior; applying both `tests/test.patch` and `solution/code.patch` should pass the target tests. diff --git a/swe-paddle/tasks/PaddlePaddle__Paddle-79633/environment/README.md b/swe-paddle/tasks/PaddlePaddle__Paddle-79633/environment/README.md new file mode 100644 index 000000000..bbb99a940 --- /dev/null +++ b/swe-paddle/tasks/PaddlePaddle__Paddle-79633/environment/README.md @@ -0,0 +1,27 @@ +# Environment Notes + +This candidate is part of the SWE-Paddle community task set. + +## Expected Environment + +- Repository: `PaddlePaddle/Paddle` +- Base commit: `58354a509a8d60b2cb3cdf6ead63a6c845eefd23` +- Resource: CPU +- GPU required: no +- Build path: Python-only source change; use a compatible installed Paddle runtime or source build with `pytest` and `httpx` available. + +## Run Order + +1. Check out `PaddlePaddle/Paddle` at the base commit. +2. Apply `tests/test.patch`. +3. Run `bash tests/test.sh`; the target behavior should fail before the fix. +4. Apply `solution/code.patch`. +5. Run `bash tests/test.sh` again; the target behavior should pass after the gold patch. + +## Minimal Test Command + +```bash +bash tests/test.sh +``` + +The verifier is responsible for deriving stable F2P and P2P node IDs from repeated runs. diff --git a/swe-paddle/tasks/PaddlePaddle__Paddle-79633/instruction.md b/swe-paddle/tasks/PaddlePaddle__Paddle-79633/instruction.md new file mode 100644 index 000000000..f182dfa11 --- /dev/null +++ b/swe-paddle/tasks/PaddlePaddle__Paddle-79633/instruction.md @@ -0,0 +1,21 @@ +# 修复大规模分布式启动时 KV server 请求阻塞 + +## 详细描述 + +Paddle 的分布式启动流程会通过 KV server 收集各节点信息。节点数量较多、多个节点同时注册时,部分 `put` 或 `get_prefix` 请求可能长时间没有响应,导致所有节点一直等待,训练任务无法正常启动。 + +此外,如果某个客户端建立连接后没有把请求发送完整,这个连接可能一直占用服务端处理能力,后续正常请求也会受到影响。 + +需要让 KV server 在并发请求和未完成请求出现时仍能继续提供服务,并确保停止服务时不会卡住或遗留监听端口。 + +## 验收说明 + +- 多个客户端同时写入并读取节点信息时,请求能够完成,记录不能丢失或相互覆盖。 +- 客户端连接后没有完成请求时,服务端应在合理时间内释放该连接,其他正常请求仍能继续处理。 +- KV server 停止时不能卡住,并应释放监听端口;已有的 `put`、`get` 和 `get_prefix` 行为保持不变。 + +## 技术要求 + +- 熟悉 Python HTTP server 和并发请求处理。 +- 熟悉 PaddlePaddle 分布式 launch 流程。 +- 熟悉网络服务的启动、停止和异常连接处理。 diff --git a/swe-paddle/tasks/PaddlePaddle__Paddle-79633/proposal.md b/swe-paddle/tasks/PaddlePaddle__Paddle-79633/proposal.md new file mode 100644 index 000000000..b4a1fe965 --- /dev/null +++ b/swe-paddle/tasks/PaddlePaddle__Paddle-79633/proposal.md @@ -0,0 +1,56 @@ +# Task Proposal: PaddlePaddle__Paddle-79633 + +## 1. 来源信息 + +- Instance ID:`PaddlePaddle__Paddle-79633` +- PR 链接:https://github.com/PaddlePaddle/Paddle/pull/79633 +- PR 标题:`[Distributed Strategy] Fix KV server hangs under concurrent requests` +- `base_commit`:`58354a509a8d60b2cb3cdf6ead63a6c845eefd23` +- merged 时间:`2026-08-10T12:30:53Z` +- 你的身份:熟悉该模块的 contributor +- 后续联系人:TBD + +## 2. 问题一句话 + +分布式启动使用的 KV server 在并发注册或遇到未完成请求时可能阻塞,导致其他节点无法继续完成启动同步。 + +## 3. 为什么适合作为 SWE-Paddle 样本 + +- **真实性**:问题来自大规模分布式启动过程中节点注册和信息同步的真实阻塞场景。 +- **代表性**:覆盖 Python 网络服务的并发处理、异常连接回收和服务生命周期管理。 +- **边界清楚**:production change 仅涉及 KV server,测试也只访问本机临时端口。 +- **非平凡性**:修复需要同时保证并发请求、半开连接和正常停止,不是简单修改返回值或错误信息。 +- **环境友好性**:原 PR 测试使用 CPU、loopback 网络和系统分配的临时端口,不需要 GPU、外部服务或数据集。 + +## 4. 任务类型和标签 + +- 任务类型:`bug_fix` +- 执行后端:`cpu` +- 设备范围:`cpu_only` +- 模块标签:`[distributed_launch, kv_server, concurrency, networking]` + +## 5. 验证思路 + +- 目标测试命令:`bash tests/test.sh` +- 目标测试文件:`test/legacy_test/test_kv_server.py` +- 修复前预期:服务停止测试通过;并发请求测试和未完成请求超时测试失败。 +- 修复后预期:三个原 PR 测试全部通过,服务可以并发响应、释放异常连接并正常停止。 +- P2P 候选:`TestKVServerStop::test_stop_is_clean_and_idempotent_state` +- F2P 候选:`TestKVServerConcurrent::test_concurrent_put_get_prefix`、`TestKVServerRequestTimeout::test_half_open_connection_is_released_after_timeout` + +## 6. 环境与资源 + +- 资源需求:CPU +- Paddle 来源:`PaddlePaddle/Paddle` source checkout at `base_commit` +- 是否能提供 Docker:暂无 +- patch 类型:Python-only +- 环境建议:使用与源码兼容的 Paddle Python 环境;测试仅使用本机 loopback 网络和临时端口。 +- 最小测试命令:`bash tests/test.sh` +- 是否有 oracle 日志:由 SWE-Paddle verifier 结果另行维护 + +## 7. 风险自查 + +- 泄露风险:instruction 只描述可观察的阻塞、恢复和停止行为,没有给出 Gold patch 的具体实现。 +- 环境风险:测试依赖 `httpx`,该依赖随 Paddle launch 环境提供;不需要外部网络。 +- flaky 风险:测试使用系统分配的临时端口,并设置有界等待;不依赖真实集群竞态。 +- 拆分风险:PR 只解决 KV server 在并发和异常连接下阻塞这一项问题,三个测试共同验证同一服务可用性契约。 diff --git a/swe-paddle/tasks/PaddlePaddle__Paddle-79633/solution/code.patch b/swe-paddle/tasks/PaddlePaddle__Paddle-79633/solution/code.patch new file mode 100644 index 000000000..710793d67 --- /dev/null +++ b/swe-paddle/tasks/PaddlePaddle__Paddle-79633/solution/code.patch @@ -0,0 +1,46 @@ +diff --git a/python/paddle/distributed/launch/utils/kv_server.py b/python/paddle/distributed/launch/utils/kv_server.py +index 279d03eba00074a8af08888fdae89ebce33a573b..ef448983764b3fddf058cbf7c695316e7738de8d 100644 +--- a/python/paddle/distributed/launch/utils/kv_server.py ++++ b/python/paddle/distributed/launch/utils/kv_server.py +@@ -15,13 +15,20 @@ + import http.server as SimpleHTTPServer + import json + import threading +-from http.server import HTTPServer ++from http.server import ThreadingHTTPServer + from multiprocessing import Process + + from .topology import SingleNodeTopology + + + class KVHandler(SimpleHTTPServer.SimpleHTTPRequestHandler): ++ # StreamRequestHandler.setup() applies this as a socket timeout, and ++ # BaseHTTPRequestHandler.handle_one_request() turns the resulting TimeoutError into a ++ # connection close. Without it a peer that connects but never finishes sending its ++ # request line blocks the handler in rfile.readline() forever; on a single-threaded ++ # server that wedges the whole KV store and no other node can ever register. ++ timeout = 30 ++ + def do_GET(self): + with self.server.kv_lock: + ret = {} +@@ -67,7 +74,18 @@ class KVHandler(SimpleHTTPServer.SimpleHTTPRequestHandler): + return + + +-class KVServer(HTTPServer): ++class KVServer(ThreadingHTTPServer): ++ # The default socketserver.TCPServer.request_queue_size is 5, i.e. listen(5). ++ # At 768 nodes every launcher polls put()/get_prefix() every 0.5s over HTTP/1.0 ++ # (one new TCP connection per request), which overflows a 5-deep accept queue. ++ # Overflowed SYNs are dropped, and because KVClient uses timeout=None the client ++ # then waits out the ~127s kernel SYN-retransmit timeout, so a few nodes can never ++ # register and sync_peers livelocks. Serve requests concurrently and give the ++ # accept queue enough room. kv is already guarded by kv_lock, so KVHandler is ++ # safe to run on multiple threads. ++ request_queue_size = 2048 ++ daemon_threads = True ++ + def __init__(self, port): + super().__init__(('', port), KVHandler) + self.kv_lock = threading.Lock() diff --git a/swe-paddle/tasks/PaddlePaddle__Paddle-79633/tests/test.patch b/swe-paddle/tasks/PaddlePaddle__Paddle-79633/tests/test.patch new file mode 100644 index 000000000..5b1cf6d10 --- /dev/null +++ b/swe-paddle/tasks/PaddlePaddle__Paddle-79633/tests/test.patch @@ -0,0 +1,221 @@ +diff --git a/test/legacy_test/test_kv_server.py b/test/legacy_test/test_kv_server.py +new file mode 100644 +index 0000000000000000000000000000000000000000..2ba5fda2e4623fbf44864932a3c37802782c35cf +--- /dev/null ++++ b/test/legacy_test/test_kv_server.py +@@ -0,0 +1,215 @@ ++# Copyright (c) 2026 PaddlePaddle Authors. All Rights Reserved. ++# ++# Licensed under the Apache License, Version 2.0 (the "License"); ++# you may not use this file except in compliance with the License. ++# You may obtain a copy of the License at ++# ++# http://www.apache.org/licenses/LICENSE-2.0 ++# ++# Unless required by applicable law or agreed to in writing, software ++# distributed under the License is distributed on an "AS IS" BASIS, ++# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ++# See the License for the specific language governing permissions and ++# limitations under the License. ++ ++import os ++import socket ++import threading ++import time ++import unittest ++from http.server import ThreadingHTTPServer ++ ++from paddle.distributed.launch.utils.kv_client import KVClient ++from paddle.distributed.launch.utils.kv_server import KVHandler, KVServer ++ ++ ++class KVServerTestBase(unittest.TestCase): ++ def setUp(self): ++ # KVClient talks to the server over httpx, which honors *_PROXY env vars. ++ # On CI a global HTTP proxy is configured, so without a bypass the ++ # loopback request gets hijacked by the proxy (which answers 500 for ++ # 127.0.0.1) instead of reaching our server. Drop the proxy vars and ++ # force a bypass for the whole test, restoring the env afterwards. ++ self._proxy_env_backup = {} ++ for var in ( ++ "http_proxy", ++ "https_proxy", ++ "all_proxy", ++ "HTTP_PROXY", ++ "HTTPS_PROXY", ++ "ALL_PROXY", ++ "no_proxy", ++ "NO_PROXY", ++ ): ++ self._proxy_env_backup[var] = os.environ.pop(var, None) ++ os.environ["no_proxy"] = "*" ++ os.environ["NO_PROXY"] = "*" ++ self.addCleanup(self._restore_proxy_env) ++ ++ def _restore_proxy_env(self): ++ for var, val in self._proxy_env_backup.items(): ++ if val is None: ++ os.environ.pop(var, None) ++ else: ++ os.environ[var] = val ++ ++ def _start_server(self): ++ # port 0 -> let the OS pick a free ephemeral port to avoid CI collisions. ++ server = KVServer(0) ++ # KVServer binds to ('', port), so server_address[0] is 0.0.0.0. That is ++ # a valid *bind* address but not a valid *connect* target on Windows ++ # (WinError 10049), so always reach the server via the loopback address. ++ self.host = "127.0.0.1" ++ self.port = server.server_address[1] ++ server.start() ++ self.addCleanup(self._safe_stop, server) ++ client = KVClient(f"127.0.0.1:{self.port}") ++ self.assertTrue( ++ client.wait_server_ready(timeout=10), "KV server never became ready" ++ ) ++ return server, client ++ ++ @staticmethod ++ def _safe_stop(server): ++ if not server.stopped: ++ server.stop() ++ ++ ++class TestKVServerConcurrent(KVServerTestBase): ++ def test_concurrent_put_get_prefix(self): ++ # HTTPMaster.sync_peers relies on many launchers concurrently doing ++ # put()/get_prefix() against a single KV store. Now that KVServer is a ++ # ThreadingHTTPServer, each connection is handled on its own thread; the ++ # shared kv dict is guarded by kv_lock. Hammer it from many threads and ++ # assert every write lands exactly once with no lost/corrupted values. ++ server, _ = self._start_server() ++ num_clients = 32 ++ ++ # Guard the concurrent thread model itself: a regression back to a ++ # single-threaded HTTPServer would reintroduce the head-of-line blocking ++ # this fix removed. ++ self.assertIsInstance(server, ThreadingHTTPServer) ++ ++ errors = [] ++ barrier = threading.Barrier(num_clients) ++ ++ def worker(idx): ++ try: ++ # All threads block here first so the puts really overlap. ++ barrier.wait(timeout=10) ++ client = KVClient(f"127.0.0.1:{self.port}") ++ self.assertTrue(client.put(f"/workers/{idx}", f"rank{idx}")) ++ except Exception as e: ++ errors.append(e) ++ ++ threads = [ ++ threading.Thread(target=worker, args=(i,)) ++ for i in range(num_clients) ++ ] ++ for t in threads: ++ t.start() ++ for t in threads: ++ t.join(timeout=30) ++ ++ self.assertEqual(errors, [], f"concurrent workers raised: {errors}") ++ ++ reader = KVClient(f"127.0.0.1:{self.port}") ++ result = reader.get_prefix("/workers") ++ self.assertIsInstance(result, dict) ++ expected = {f"/workers/{i}": f"rank{i}" for i in range(num_clients)} ++ self.assertEqual(result, expected) ++ ++ ++class TestKVServerRequestTimeout(KVServerTestBase): ++ def setUp(self): ++ super().setUp() ++ # The production timeout is 30s; override it to keep the test fast while ++ # still exercising the exact same StreamRequestHandler.setup() -> ++ # handle_one_request() timeout-to-close path. ++ self._orig_timeout = KVHandler.timeout ++ KVHandler.timeout = 0.5 ++ self.addCleanup(self._restore_timeout) ++ ++ def _restore_timeout(self): ++ KVHandler.timeout = self._orig_timeout ++ ++ def test_half_open_connection_is_released_after_timeout(self): ++ # Guard the production configuration: a None/unset handler timeout is ++ # exactly the bug (rfile.readline() would block forever), so the value ++ # must stay a positive number even though we shrink it here for speed. ++ self.assertIsNotNone( ++ self._orig_timeout, "KVHandler.timeout must be configured" ++ ) ++ self.assertGreater(self._orig_timeout, 0) ++ ++ # A peer that connects but never finishes sending its request line used ++ # to wedge the handler forever in rfile.readline(). With the socket ++ # timeout the handler must give up, close the connection, and free its ++ # worker thread. Observable contract: the client sees EOF shortly after ++ # the timeout, and the server keeps serving other requests. ++ server, client = self._start_server() ++ ++ sock = socket.create_connection((self.host, self.port), timeout=10) ++ self.addCleanup(sock.close) ++ try: ++ start = time.time() ++ sock.settimeout(10) ++ # Send nothing: leave the request line unfinished on purpose. ++ data = sock.recv(64) ++ elapsed = time.time() - start ++ finally: ++ sock.close() ++ ++ # Server-side timeout closed the connection -> clean EOF, not a hang. ++ self.assertEqual( ++ data, b"", "server did not close the stalled connection" ++ ) ++ self.assertLess( ++ elapsed, ++ 5, ++ "stalled connection was not released promptly after timeout", ++ ) ++ ++ # The stalled connection must not have wedged the store: a normal ++ # request still succeeds (proves the worker thread was freed and the ++ # server is still accepting). ++ self.assertTrue(client.put("/after_timeout", "ok")) ++ self.assertEqual(client.get("/after_timeout"), "ok") ++ ++ ++class TestKVServerStop(KVServerTestBase): ++ def test_stop_is_clean_and_idempotent_state(self): ++ server, client = self._start_server() ++ self.assertTrue(server.started) ++ self.assertFalse(server.stopped) ++ # Sanity: it is actually serving before we stop it. ++ self.assertTrue(client.put("/k", "v")) ++ self.assertEqual(client.get("/k"), "v") ++ ++ # stop() must shut down serve_forever, join the listener thread and ++ # close the socket without hanging. ++ stop_done = threading.Event() ++ ++ def _stop(): ++ server.stop() ++ stop_done.set() ++ ++ stopper = threading.Thread(target=_stop) ++ stopper.start() ++ stopper.join(timeout=15) ++ self.assertTrue(stop_done.is_set(), "server.stop() hung") ++ ++ self.assertTrue(server.stopped) ++ self.assertFalse(server.listen_thread.is_alive()) ++ ++ # The port must be released so it can be bound again. ++ rebind = socket.socket(socket.AF_INET, socket.SOCK_STREAM) ++ rebind.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1) ++ try: ++ rebind.bind((self.host, self.port)) ++ finally: ++ rebind.close() ++ ++ ++if __name__ == '__main__': ++ unittest.main() diff --git a/swe-paddle/tasks/PaddlePaddle__Paddle-79633/tests/test.sh b/swe-paddle/tasks/PaddlePaddle__Paddle-79633/tests/test.sh new file mode 100644 index 000000000..c3e6a45f6 --- /dev/null +++ b/swe-paddle/tasks/PaddlePaddle__Paddle-79633/tests/test.sh @@ -0,0 +1,4 @@ +#!/usr/bin/env bash + +set -euo pipefail +python -m pytest test/legacy_test/test_kv_server.py -q