Skip to content

test(spawn-maxbuf): widen 'yes is killed' wall-clock budget on debug, unquarantine - #36782

Merged
Jarred-Sumner merged 1 commit into
mainfrom
farm/21294e4f/spawn-maxbuf-debug-timing
Aug 2, 2026
Merged

Jarred-Sumner merged 1 commit into
mainfrom
farm/21294e4f/spawn-maxbuf-debug-timing

Conversation

@robobun

@robobun robobun commented Aug 2, 2026

Copy link
Copy Markdown
Collaborator

test/js/bun/spawn/spawn-maxbuf.test.ts has been whole-file [ FLAKY ] in test/expectations.txt, hiding all 16 maxBuffer/timeout cases on every CI lane. Probed in build 87834 it passes every release and release-asan lane cleanly, so the quarantine is stale for CI. What it was hiding is a debug-only false positive: under bun bd the two yes is killed cases fail deterministically.

Repro

$ bun bd test test/js/bun/spawn/spawn-maxbuf.test.ts -t "yes is killed"
(fail) yes is killed > Bun.spawn
  error: expect(received).toBeLessThan(expected)
  Expected: < 100
  Received: 170
(fail) yes is killed > Bun.spawnSync
  error: expect(received).toBeLessThan(expected)
  Expected: < 100
  Received: 162

Cause

Both tests assert Date.now() from before spawn() to after exited is under 100ms, with the comment "make sure it's not waiting a full tick". The child is [bunExe(), "exec", "yes"], so the measured window includes the child bun process reaching main() before yes can write its first byte. On a debug+ASAN build that startup alone is ~150-250ms:

$ bun bd -e 'const t=Date.now(); Bun.spawnSync([process.execPath,"-e",""]); console.log(Date.now()-t)'
185

so the 100ms budget is exhausted before maxBuffer has anything to react to. Release and release-asan start in <15ms and stay well inside the bound.

Fix

Widen the wall-clock window to isDebug ? 1000 : 100, matching the pattern used elsewhere for timing budgets that include a debug-build spawn (e.g. fetch-tls-abortsignal-timeout.test.ts, buffer-indexof-worstcase.test.ts). Release keeps the original 100ms bound unchanged. Promptness of the kill itself is already asserted byte-precisely by the "caps the buffer while the child is still writing" tests (stdout.length <= maxBuffer + 64*1024 against an 8MB firehose), so this assertion is only the coarse "not deferred by seconds" sanity check.

Drop the whole-file quarantine line.

Verification

$ bun bd test test/js/bun/spawn/spawn-maxbuf.test.ts
 16 pass
 0 fail

Five consecutive bun bd runs: yes is killed at 128-157ms (< 1000). Release: 2-11ms (< 100).

Test-only change; no runtime behaviour is touched.


no test proof · iteration 0 · test-only change (timing threshold + unquarantine), no src/ change to stash

…builds

The two 'yes is killed' cases time from before spawn() to after exited,
which includes the child process's startup. The child is bunExe() itself
(for cross-platform `yes` via `bun exec`), and a debug+ASAN bun takes
~150-250ms to reach main(), so the 100ms ceiling is missed before the
child has written a byte. Release and release-asan start in <15ms and
pass.

Widen the window to 1000ms under isDebug (release stays at 100ms). The
byte-bound 'caps the buffer' tests already prove maxBuffer stops reading
within one 64KB read of the limit, so this assertion is only the coarse
'kill wasn't deferred' check.

Drop the whole-file [FLAKY] quarantine: the unmodified file already
passes every release and release-asan CI lane (build 87834), and with
this change `bun bd test` passes locally too.
@coderabbitai

coderabbitai Bot commented Aug 2, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro

Run ID: d0ca4433-435c-4482-bff9-0e813eb04bfc

📥 Commits

Reviewing files that changed from the base of the PR and between fdbaf06 and c41ec15.

📒 Files selected for processing (2)
  • test/expectations.txt
  • test/js/bun/spawn/spawn-maxbuf.test.ts
💤 Files with no reviewable changes (1)
  • test/expectations.txt

Walkthrough

The spawn max-buffer test now uses a longer termination window for debug builds and a shorter window for release builds. The test is also removed from the quarantine list.

Changes

Spawn max-buffer test

Layer / File(s) Summary
Build-dependent termination timing
test/js/bun/spawn/spawn-maxbuf.test.ts, test/expectations.txt
The test imports isDebug, defines a conditional killWindow, and applies it to asynchronous and synchronous timing assertions. The quarantine entry is removed.

Possibly related PRs

Suggested reviewers: jarred-sumner

🚥 Pre-merge checks | ✅ 4
✅ Passed checks (4 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly summarizes the debug timing-budget change and removal of the flaky-test quarantine.
Description check ✅ Passed The description explains the change, cause, fix, and verification results, covering the required template information.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

Comment @coderabbitai help to get the list of available commands.

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM — test-only de-flake that branches the timing budget on isDebug and restores 16 quarantined cases to CI.

What was reviewed:

  • isDebug is exported from harness.ts and the branch-per-build-type pattern matches existing usage; release keeps the original 100ms bound.
  • The property under test (kill isn't deferred by a full tick) is preserved; byte-level promptness is separately covered by the "caps the buffer" tests, so widening the coarse wall-clock check on debug loses no precision.
  • Checked the other timing assertions in the file (timeout kills at <200ms, timeout Infinity at <1500ms) — those are parent-side timers not gated on child startup writing bytes, and the PR's bun bd run confirms all 16 pass on debug.
Extended reasoning...

Overview

Two-file test-only change: (1) test/js/bun/spawn/spawn-maxbuf.test.ts widens the yes is killed wall-clock assertion from a hardcoded 100ms to isDebug ? 1000 : 100, replacing a // TODO with a comment explaining why; (2) test/expectations.txt drops the whole-file [ FLAKY ] quarantine that was hiding all 16 maxBuffer/timeout cases on every CI lane.

Security risks

None. No runtime code is touched; this only adjusts a test timing threshold and removes a quarantine entry.

Level of scrutiny

Low. This is the canonical de-flake pattern the repo's review guidance calls for: "When de-flaking, keep asserting the property the original assertion protected — branch per-platform rather than dropping precision." The release bound is unchanged at 100ms, and the debug-only widening is well-justified (the measured window includes a debug+ASAN child bun reaching main(), ~150-250ms on its own). The added comment names the mechanism and points to the byte-precise sibling tests that guard actual promptness.

Other factors

  • The PR description shows a probe build (87834) passing cleanly on release and release-asan, five consecutive bun bd runs at 128-157ms (well under 1000), and release at 2-11ms (well under 100), so the new bounds have headroom on both sides.
  • Unquarantining is a net coverage gain — the file had been entirely skipped on all lanes.
  • I spot-checked the remaining timing assertions in the file for the same debug-startup hazard: the timeout: 100 / < 200 cases fire a parent-side timer and send the signal regardless of child readiness (default SIGHUP disposition terminates during startup), and the bun bd 16-pass verification confirms they hold.
  • No prior reviews or comments on the timeline.

@Jarred-Sumner
Jarred-Sumner merged commit 862a285 into main Aug 2, 2026
52 of 53 checks passed
@Jarred-Sumner
Jarred-Sumner deleted the farm/21294e4f/spawn-maxbuf-debug-timing branch August 2, 2026 14:25
@robobun

robobun commented Aug 2, 2026

Copy link
Copy Markdown
Collaborator Author

Follow-up: #36792 fixes the remaining timeout kills the process > Bun.spawnSync boundary flake (toBeGreaterThan(100) receiving exactly 100) that surfaced once this file was unquarantined.

Jarred-Sumner pushed a commit that referenced this pull request Aug 2, 2026
…wnSync timeout (#36792)

Follow-up to #36782, which unquarantined `spawn-maxbuf.test.ts` and
exposed a latent boundary flake in a different subtest.

## Failure

```
240 |     expect(timeEnd - timeStart).toBeGreaterThan(100); // make sure it actually waits
error: expect(received).toBeGreaterThan(expected)
Expected: > 100
Received: 100
✗ timeout kills the process > Bun.spawnSync [101.64ms]
```

Seen on main [build 87859](https://buildkite.com/bun/bun/builds/87859)
(alpine 3.23 x64, retry passed) and [build
87890](https://buildkite.com/bun/bun/builds/87890) (debian 13 aarch64,
red on every retry). The assertion has been in the file since #18293; it
was hidden by the whole-file `[ FLAKY ]` quarantine until #36782 removed
it.

## Cause

The `spawnSync` timeout deadline is computed as `CLOCK_MONOTONIC +
100ms` at nanosecond precision (`js_bun_spawn_bindings.rs`:
`now.add_ms(timeout_ms)`), and the timeout fires when `CLOCK_MONOTONIC
>= deadline` (`SpawnSyncEventLoop::tick_with_timeout`). So by the time
`spawnSync` returns, at least 100ms of monotonic time has elapsed.

The test measures with `Date.now()`, which truncates the wall clock to
whole milliseconds. For a real elapsed interval of 100.x ms, `floor(end)
- floor(start)` can be exactly 100 (e.g. start at T+0.1ms →
`Date.now()=T`, end at T+100.2ms → `Date.now()=T+100`). The test
duration `[101.64ms]` in the CI output confirms the wait itself was not
short.

Local 500-iteration probe on release:

```
{"min":100,"hit100":7,"under100":0,"iterations":500}
```

The delta lands on exactly 100 in ~1.4% of runs and never goes below
100, so `>=` is the tight bound.

## Fix

Change `toBeGreaterThan(100)` to `toBeGreaterThanOrEqual(100)`. The
lower bound is kept because it is the only thing in this test that would
catch `timeout: 100` being misread as `timeout: 0` (the
`exitedDueToTimeout`/`signalCode` checks would still pass if the kill
fired immediately). Node's own `test-child-process-spawnsync-timeout.js`
only asserts the upper bound and relies on `error.code === 'ETIMEDOUT'`;
Bun's `exitedDueToTimeout` is equivalent, but the `>= 100` here is cheap
extra coverage.

The sibling `toBeGreaterThan(1000)` assertions in the `timeout Infinity`
block measure a child `sleep 1` plus `bunExe()` startup, which is always
comfortably above 1000ms, so they are left as-is.

## Verification

```
$ bun bd test test/js/bun/spawn/spawn-maxbuf.test.ts
 16 pass
 0 fail
```

Five consecutive full-file debug runs pass.

Test-only change; no runtime behaviour is touched.

<!-- robobun:evidence:begin -->

---

**no test proof** · iteration 0 · Platform-specific test-only change;
deferring to CI.

<!-- robobun:evidence:end -->
Jarred-Sumner pushed a commit that referenced this pull request Aug 3, 2026
Empirically re-derived which `test/expectations.txt` entries are still
needed by removing all 29 and running the full CI matrix ([build
87834](https://buildkite.com/bun/bun/builds/87834)).

## Result: 29 entries → 3

### Kept (3): still fail on the named lane

| entry | lane | observed failure (build 87834) |
|---|---|---|
| `test/bundler/native-plugin.test.ts` | WINDOWS | MSB8020: ClangCL
build tools not found (agent image gap) |
| `test/js/node/test/parallel/test-net-pingpong.js` | WINDOWS |
named-pipe half-close: count 1000 !== 1001 |
| `test/js/node/test/sequential/test-net-listen-shared-ports.js` | LINUX
| SO_REUSEPORT shared-listener semantics; passes on macOS/Windows |

### Deleted (6): vendored Node tests that fail deterministically on
every lane

These can never pass as vendored; removing the files instead of
re-quarantining.

| file | reason |
|---|---|
| `test-stream-wrap.js`, `test-stream-wrap-drain.js`,
`test-stream-wrap-encoding.js` | require `internal/js_stream_socket`
which Bun does not implement |
| `test-net-connect-keepalive.js`, `test-net-server-keepalive.js` |
assert `_handle.setKeepAlive` receives seconds (libuv convention); Bun's
`_handle` is Bun.Socket (ms). End-to-end TCP_KEEPIDLE coverage is in
`test/js/bun/net/socket.test.ts` |
| `test-set-http-max-http-headers.js` | spawns
`test-http-max-http-headers.js` which is not vendored |

### Moved to `no-validate-exceptions.txt` (7): fail only via
unchecked-exception assertions

These now **run** on ASAN with `validateExceptionChecks` off, instead of
being removed from the run entirely.

| file | unchecked exception scope |
|---|---|
| `test/integration/next-pages/test/dev-server-ssr-100.test.ts` |
`JSOrderedHashTable::getImpl` → `executeBoundCall` |
| `test/integration/next-pages/test/dev-server.test.ts` | same |
| `test/integration/next-pages/test/next-build.test.ts` | same |
| `test/js/third_party/next-auth/next-auth.test.ts` | same |
| `test/napi/napi.test.ts` | `Process_functionDlopen`
(BunProcess.cpp:397) |
| `test/cli/run/require-cache.test.ts` | `NapiClass::finishCreation`
(NapiClass.cpp:120) |
| `test/cli/inspect/inspect.test.ts` | `getOwnNonIndexPropertyNames` →
`JSObjectInlines::get` (inspector Runtime.evaluate) |

Also bumped `esm-fixture-leak-small.mjs` ASAN threshold 400→500 MB
(build 87834 measured 407 MB; ASAN quarantine overhead) so
`require-cache.test.ts` passes end to end on ASAN.

### Removed (13): now pass on their named lane

| entry | was scoped to | now passes on |
|---|---|---|
| `test/js/node/test/parallel/test-repl-close.js` | WINDOWS-AARCH64 |
windows 11 aarch64 |
| `test/js/node/test/parallel/test-tls-connect-memleak.js` |
LINUX-X64-MUSL | alpine 3.23 x64 + aarch64 |
| `test/js/bun/spawn/spawn-maxbuf.test.ts` | (all) | every lane (also
fixed for debug in #36782) |
| `test/js/bun/spawn/spawn.test.ts` | ASAN | every lane
(heap-use-after-free fixed in #36783) |
| `test/js/sql/tls-sql.test.ts` | ASAN | debian 13 x64-asan |
| `test/js/node/url/pathToFileURL.test.ts` | ASAN | debian 13 x64-asan |
| `test/js/node/fs/abort-signal-leak-read-write-file.test.ts` | ASAN |
debian 13 x64-asan |
| `test/js/web/streams/streams-leak.test.ts` | ASAN | debian 13 x64-asan
|
| `test/js/node/test/parallel/test-net-server-listen-path.js` | WINDOWS
| windows 2019 x64 + 11 aarch64 |
| `test/js/node/test/parallel/test-net-pipe-connect-errors.js` | WINDOWS
| fixed in #36786 |
| `test/js/node/test/parallel/test-net-client-bind-twice.js` | WINDOWS |
fixed in #36786 |
| `test/js/node/test/parallel/test-net-server-reset.js` | WINDOWS |
fixed in #36786 |
| `test/js/bun/io/fetch/fetch-abort-slow-connect.test.ts` | DARWIN |
darwin 26 aarch64 + 14 x64 |

### Caveats

- `test-tls-connect-memleak.js` and `fetch-abort-slow-connect.test.ts`
were `FLAKY` quarantines; both passed in probe build 87834 and
confirmation builds 87845 / 87860 / 87868.

<!-- robobun:evidence:begin -->

---

**no test proof** · iteration 4 · docs-only change; test-proof not
applicable

<!-- robobun:evidence:end -->
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants