Skip to content

test(server): run the real Hub project-packages suite in its own Bun process - #215

Merged
cjhyy merged 1 commit into
mainfrom
codex/test/hub-project-packages-flake-20261022
Oct 22, 2026
Merged

cjhyy merged 1 commit into
mainfrom
codex/test/hub-project-packages-flake-20261022

Conversation

@cjhyy

@cjhyy cjhyy commented Oct 22, 2026

Copy link
Copy Markdown
Owner

Problem

packages/server/src/panels/project-packages.test.ts ("real Hub HTTP serves and executes each project's retained package…") failed intermittently in tests (rest). The task job failed with one of three errors:

  • process is unavailable or belongs to another Panel App
  • process stdin is unavailable
  • Tool did not return a completed result

Root cause (runner evidence)

  • Alone: 50/50 passes on the runner (2 jobs × 25 runs).
  • In the full rest shard: failed in 3/8 runs (diag run 39713091209) and 1/10 runs (diag run 39714111699).
  • Instrumentation: I added logging to the executor and process service. In each failure, the fixture tool's output and exit arrived before the Host had ended stdin. In some runs the child error event was EBADF: bad file descriptor, epoll_ctl, with exit(code=null) and the parent's stdin closed, while the real child kept running and printed its result later.
  • strace (close, epoll_ctl, pidfd_open, socketpair, vfork): late in the shard the single Bun process holds about 1400–4500 descriptors. Inside Bun's synchronous spawn path, the main thread does this:
    pidfd_open(8734) = 4365<pidfd>
    close(4362<socket>) close(4364<socket>)   # child ends, normal
    close(4365<pidfd>) = 0                    # closes the pidfd it just opened
    close(4365) = -1 EBADF  (x2)
    epoll_ctl(ADD, 4365) = -1 EBADF           # Bun reports the spawn as failed
    
    A spawn that succeeded a few seconds earlier, with the same fd numbers, shows no such close. In the other variant, a parent stdio socket is closed under the child: the fixture reads EOF, answers, and exits before processes.end().
  • Not specific to this suite: the same EBADF ... epoll_ctl also failed an unrelated Bun.spawn test in the shard (hub-automations.test.ts lease test).

The product code is not involved. Hub and Desktop run on Node, and the executor and process service behave correctly when their descriptors are intact. Recent Bun PRs fix similar double-close and fd-confusion bugs in spawn and IO (oven-sh/bun#38354, #43626, #43814). Upgrading Bun is a separate, repo-wide decision.

Fix

  • The rest shard now ignores **/server/src/panels/project-packages.test.ts.
  • A new rest step runs that suite in its own Bun process with the same 30s timeout. The shard already uses this pattern for other suites that a long shared Bun process destabilizes.

Verification

  • bun run lint:workflow-test-paths passes.
  • Running the panels directory with the ignore pattern excludes the suite.
  • Runner verification (diag run 39719250655): 10 × (rest shard with the ignore + this suite run 5 times alone). Results are in the PR comments.

Follow-ups (not in this PR)

  • The rest-shard Bun process climbs to about 4500 open descriptors (pidfd numbers reach 4550). Something in the shard leaks fds.
  • The same Bun spawn defect can still hit other spawn-heavy tests in the shard.

🤖 Generated with Claude Code

…process

project-packages.test.ts failed intermittently in `tests (rest)` with three
different job errors ("process is unavailable or belongs to another Panel
App", "process stdin is unavailable", "Tool did not return a completed
result"). It passed 50/50 when run alone on the runner and failed in 3/8
and 1/10 full rest-shard runs.

Root cause, from instrumented runner runs with strace on close/epoll_ctl/
pidfd_open: late in the shard the single Bun process holds 1400-4500 open
descriptors and has spawned hundreds of children. Inside the synchronous
spawn path, Bun 1.3.11 on Linux closes the pidfd it has just opened
(pidfd_open = N, close(N) = 0, close(N) = EBADF twice) and then
epoll_ctl(ADD, N) fails with EBADF. Bun emits "EBADF: bad file descriptor,
epoll_ctl" as a child 'error' plus an exit with code null and closes the
stdio pipes while the real child keeps running; the executor's next
stdin write/end then finds the process gone. In the other variant a parent
stdio socket is closed under the child, so the fixture tool reads EOF and
answers before the Host has ended stdin. The same EBADF also failed an
unrelated Bun.spawn test in the shard (hub-automations lease test).

Product code is not involved: Hub and Desktop run on Node, and the
executor and process service behave correctly with intact descriptors.
Run this suite in its own Bun process, as the shard already does for
other suites that a long shared Bun process destabilizes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@cjhyy

cjhyy commented Oct 22, 2026

Copy link
Copy Markdown
Owner Author

Runner verification (workflow_dispatch run 39719250655, diag branch): 10 parallel jobs, each running the full rest shard with the new ignore and then this suite alone 5 times. All 10 shards: 5625 tests, 0 failures. Isolated suite: 50/50 passes. Before the fix, the same rest shard failed this suite in 3/8 (run 39713091209) and 1/10 (run 39714111699) runs.

@cjhyy
cjhyy merged commit 8ed4756 into main Oct 22, 2026
12 checks passed
@cjhyy
cjhyy deleted the codex/test/hub-project-packages-flake-20261022 branch October 22, 2026 05:34
cjhyy added a commit that referenced this pull request Oct 22, 2026
gitOps.lsremote's "returns the HEAD sha" test failed in core-rest five
times since 2026-10-21 (PR #214 among them, plus two main runs) with
only "Expected: true, Received: false". Instrumented on the runner, the
failure is `git ls-remote ... exited null`: git was killed by SIGPIPE
3-7ms after spawn, with empty stdout and stderr and no process.kill
aimed at it. GIT_TRACE_PACKET shows the whole ls-remote exchange
completed, including the final flush to upload-pack, so git died on its
first write to its own stdout: the parent's read end was already gone.
A plain node:child_process spawn of the same command fails the same way,
so safeSpawn is not involved.

This is the Bun 1.3.11 Linux bug already isolated in #215/#216: late in
a long shard, Bun closes a just-created child descriptor during spawn.
Measured on the runner: 9 failures in about 13,900 calls inside the
core-rest shard, 0 in about 7,800 calls with the suite run alone. Run it
alone, like knownMarketplaces, and compare the whole result so a future
failure prints git's error rather than a bare false.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant