Skip to content

ci: regenerate parallel-allowlist from post-#36175 build data - #36585

Merged
Jarred-Sumner merged 1 commit into
mainfrom
farm/8d506697/parallel-allowlist-exclude-heapstats-tests
Jul 31, 2026
Merged

Jarred-Sumner merged 1 commit into
mainfrom
farm/8d506697/parallel-allowlist-exclude-heapstats-tests

Conversation

@robobun

@robobun robobun commented Jul 31, 2026 •

Copy link
Copy Markdown
Collaborator

Problem

The parallel-batch allowlist (test/parallel-allowlist.json) was generated from builds 82146-83745, which predate #36175 (the PR that introduced the parallel batch). The generator's rule is "zero flaky/failed annotations in the scanned builds", but at that point no test had ever run inside a shared bun test --parallel process, so the allowlist couldn't know which tests are incompatible with in-process parallelism.

The result: tests that measure process-wide state (heapStats().objectTypeCounts, rss(), expectMaxObjectTypeCount) or have tight wall-clock budgets were included, and they now flake on nearly every build. Across the last 400 completed PR builds, the top "failed in the parallel batch; passed alone" annotations by distinct build count:

test builds (of 400)
test/js/web/streams/streams-leak.test.ts 348
test/js/bun/shell/leak.test.ts 340
test/js/bun/net/socket-retention.test.ts 334
test/js/web/fetch/fetch-tls-abortsignal-timeout.test.ts 282
test/js/bun/spawn/spawn-streaming-stdout.test.ts 267
test/regression/issue/08965/08965.test.ts 261
test/regression/issue/20875.test.ts 151
test/js/node/http/node-http-pinned-write.test.ts 110
test/regression/issue/27272.test.ts 103
test/js/third_party/esbuild/esbuild-child_process.test.ts 103

Each of these generates a yellow annotation + serial retry on most builds.

Fix

Re-ran scripts/update-parallel-allowlist.mjs (300 PR builds, 85133-86444). 609 distinct files flaked or failed in that window; the generator drops them per its existing rule.

  • 282 dirs -> 215
  • 1642 files -> 1296 eligible for the batch
  • 153 -> 160 explicit excludes

All six of the top offenders above are now excluded (either via excludeFiles or their dir dropped).

Verification

streams-leak.test.ts asserts rss() < 250MB; in the batch it sees 390MB-2.6GB from neighbouring tests. shell/leak.test.ts's #11816 tests assert heapStats().objectTypeCounts.ShellInterpreter <= 3 in-process; any concurrent test using $ contributes. socket-retention.test.ts and spawn-streaming-stdout.test.ts use expectMaxObjectTypeCount similarly. These are correct tests in an isolated process and cannot be made parallel-safe without moving the assertions into a subprocess; excluding them from the batch is what the generator already does once it has data.


no test proof · iteration 0 · docs-only change; test-proof not applicable

The allowlist was generated from builds 82146-83745, which predate the
parallel-batch feature itself (#36175). Tests that measure process-wide
state (heapStats().objectTypeCounts, rss(), expectMaxObjectTypeCount) or
have tight timing budgets were included because they had never run
in-process alongside other test files before.

Across 400 recent builds the top parallel-batch-only flakes ("passed
alone" after failing in the batch) are:

  streams-leak.test.ts                  348/400 builds
  shell/leak.test.ts                    340/400
  socket-retention.test.ts              334/400
  fetch-tls-abortsignal-timeout.test.ts 282/400
  spawn-streaming-stdout.test.ts        267/400
  08965.test.ts                         261/400

Regenerated with scripts/update-parallel-allowlist.mjs from builds
85133-86444 (300 PR builds, post-feature). 282 dirs -> 215, 1642 files
-> 1296 in the batch, 153 -> 160 explicit excludes. The generator
already filters on "zero flaky/failed annotations in the window"; the
previous run simply had no data to filter on.
@coderabbitai

coderabbitai Bot commented Jul 31, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro

Run ID: 879b9d50-609e-4f2f-94a7-6294d4567fd8

📥 Commits

Reviewing files that changed from the base of the PR and between f68e504 and 08b90a6.

📒 Files selected for processing (1)
  • test/parallel-allowlist.json

Walkthrough

The generated parallel-test configuration now contains updated build metadata, directory allowlists, and excluded-test files across Bun, Node.js, third-party, web, and regression suites.

Changes

Parallel test allowlist

Layer / File(s) Summary
Regeneration metadata and root coverage
test/parallel-allowlist.json
The generation metadata, scanned build range, aggregate counts, and initial directory coverage were updated.
Parallel directory coverage
test/parallel-allowlist.json
The allowlist was recalculated across Bun, Deno, first-party, Node.js, third-party, web, N-API, and regression test directories.
Excluded-test inventory
test/parallel-allowlist.json
The excluded-test list was replaced with a new inventory covering multiple test suites.

Possibly related PRs

  • oven-sh/bun#35944: The regenerated exclusions may include parallel-safe test exclusions addressed by this PR.

Suggested reviewers: jarred-sumner

🚥 Pre-merge checks | ✅ 4
✅ Passed checks (4 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly states that the parallel allowlist was regenerated from post-#36175 build data.
Description check ✅ Passed The description explains the problem, fix, data changes, affected tests, and verification evidence.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

Comment @coderabbitai help to get the list of available commands.

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM — mechanical regeneration of the parallel-batch allowlist from newer CI data.

What was reviewed:

  • Consumer in scripts/runner.node.mjs:388-394 — the new "" dir entry is handled correctly (matches top-level package-json-lint.test.ts only).
  • Self-test test/internal/parallel-allowlist.test.ts invariants: all 160 excludeFiles exist on disk and have their dirname in dirs; counts match _meta.stats (215/160).
  • All ten top-offender tests from the PR description are excluded (either in excludeFiles or their dir dropped).
Extended reasoning...

Overview

Single-file change to test/parallel-allowlist.json, a generated CI configuration file that tells scripts/runner.node.mjs which test files may run in the shared bun test --parallel batch. The generator script (scripts/update-parallel-allowlist.mjs) is unchanged — only its output is refreshed against builds 85133-86444, which now include real parallel-batch flake data (the previous window predated the batch's introduction in #36175).

Net effect: 282→215 dirs, 153→160 explicit excludes, 1642→1296 batch-eligible files. Tests dropped from the batch fall back to serial execution — no coverage is lost.

Security risks

None. This is CI test-scheduling metadata; it does not touch runtime, build, or any shipped code path.

Level of scrutiny

Low. The failure mode of a wrong entry is a yellow flake annotation + serial retry (the exact symptom this PR is fixing), not a broken build or missed test. I verified structural correctness against both the runner's consumer logic and the repo's own self-test:

  • jq counts match _meta.stats exactly (215 dirs, 160 excludes).
  • Every excludeFiles entry exists on disk and its dirname() is present in dirs (so test/internal/parallel-allowlist.test.ts passes).
  • The new "" dir entry is intentional: isParallelAllowlisted uses slash === -1 ? "" : path.slice(0, slash), so it matches only top-level test files — currently just package-json-lint.test.ts.
  • Spot-checked all ten top-offender tests from the PR table: each is now excluded either via excludeFiles (streams-leak, shell/leak, socket-retention, fetch-tls-abortsignal-timeout, 20875, node-http-pinned-write, 27272) or by dropping their dir (js/bun/spawn, regression/issue/08965, js/third_party/esbuild).

Other factors

The PR description's mechanism analysis (RSS/heapStats assertions being process-wide) is sound and matches why these tests can't be made parallel-safe without subprocess isolation. This is exactly the kind of periodic data refresh the generator was built for.

@github-actions

Copy link
Copy Markdown
Contributor

Found 3 issues this PR may fix:

  1. node-net.test.ts: un-awaited expectMaxObjectTypeCount in named-pipe test fails unrelated subsequent tests on Windows #32092 - PR removes js/node/net from the parallel-safe dirs, preventing the un-awaited expectMaxObjectTypeCount in node-net.test.ts from contaminating subsequent tests in the batch
  2. ASAN CI: JSC assertion in JSObject::getOwnPropertyDescriptor during worker terminate (test-worker-message-port-transfer-terminate) #34095 - PR removes js/node/worker_threads from the parallel-safe dirs, reducing the concurrency that triggers the intermittent JSC assertion failure during worker terminate
  3. ASAN CI: ExceptionScope::assertNoException during worker terminate (worker-transfer-terminate-stress, separate from #34095) #34690 - Same as above — removing js/node/worker_threads from parallel batches mitigates the timing-sensitive ASAN ExceptionScope::assertNoException race during worker termination

If this is helpful, copy the block below into the PR description to auto-close these issues on merge.

Fixes #32092
Fixes #34095
Fixes #34690

🤖 Generated with Claude Code

@robobun

robobun commented Jul 31, 2026

Copy link
Copy Markdown
Collaborator Author

Re the find-issues bot: none of #32092, #34095, #34690 are fixed by this PR.

Leaving the PR description as-is.

@Jarred-Sumner
Jarred-Sumner merged commit 30032c6 into main Jul 31, 2026
53 checks passed
@Jarred-Sumner
Jarred-Sumner deleted the farm/8d506697/parallel-allowlist-exclude-heapstats-tests branch July 31, 2026 23:42
@robobun

robobun commented Aug 4, 2026

Copy link
Copy Markdown
Collaborator Author

Heads up: the regen dropped js/bun/util/inspect-error-leak.test.js from excludeFiles because the scanned window (85133-86444) only contained builds where it was already excluded, so the generator saw zero flakes. Once included it times out on x64-asan and crashes the parallel worker on Windows on most builds since 86456. Re-added in #36867.

robobun added a commit that referenced this pull request Aug 4, 2026
update-parallel-allowlist.mjs rebuilds excludeFiles from scratch based
on flake annotations in the scanned build window. A file that is already
excluded runs serially, produces no annotation, and so gets dropped on
the next regen, which is how #36585 lost inspect-error-leak. Add a guard
that fails CI if a regen drops any of the known in-process RSS/process-
state leak tests, so the oscillation is caught at the regen PR rather
than as a downstream flake.
robobun added a commit that referenced this pull request Aug 16, 2026
…e-exclude streams-leak

scripts/update-parallel-allowlist.mjs rebuilds excludeFiles from scratch
from flake annotations in the scanned build window. A file that is
already excluded runs serially, produces no annotation, and so is dropped
on the next regen, then flakes once re-included. #36585 dropped
inspect-error-leak this way (flaked on nearly every build from 86456 until
the #39236 regen re-excluded it); #39236 in turn dropped streams-leak,
which has flaked in the parallel batch on every macOS lane of every main
build since (98887 onward).

Add a guard in test/internal/parallel-allowlist.test.ts listing the tests
that assert on in-process rss()/heapStats() and so cannot be batch-safe;
a regen that drops one of them fails there, naming the file. Re-add
streams-leak to excludeFiles so the guard passes on this tree.
robobun added a commit that referenced this pull request Aug 16, 2026
handle-leak, sourcetextmodule-leak and spawn-pipe-leak each went through a
full turn of the regen loop already (excluded by #36175, dropped by
#36585, re-excluded by #39236); zlib/leak was first excluded by #39236
and is next in line. All four measure rss() in the test process itself.
robobun added a commit that referenced this pull request Aug 17, 2026
scripts/update-parallel-allowlist.mjs rebuilds excludeFiles from scratch
from the flake annotations in its scan window. A file that is already
excluded runs serially, never produces a batch annotation, and so is
dropped by the next regen, then flakes once it is back in the batch.
#36585 dropped inspect-error-leak and sourcetextmodule-leak this way and
both have flaked in the batch on most builds since (x64-asan timeouts,
Windows worker exit 9); #39236 re-excluded them, dropped streams-leak
instead, and has now been reverted (#39369), putting the first two back
in the batch.

Add a guard in test/internal/parallel-allowlist.test.ts listing the
dedicated leak tests that assert on rss()/heapStats() inside the test
process; a regen that drops one of them fails there naming the file.
Re-add the three pinned files that the current table runs in the batch
(inspect-error-leak, sourcetextmodule-leak, zlib/leak) so the guard
passes on this tree.
robobun added a commit that referenced this pull request Aug 17, 2026
… more in-process leak tests

The runner reads only parallel-allowlist.json, so a denylist entry keeps a
file out of the batch only once a regen has copied it into excludeFiles.
Assert that every denylisted file in a listed dir is in excludeFiles (and
that denylist entries are real files), so a by-hand denylist addition or a
table produced by an older script fails in CI instead of running the file
in the batch. Against the table from before #39373 the check names the 97
files that PR's body describes as batch-flaking.

Add the three in-process RSS/heapStats leak tests whose batch failures
predate the window the denylist was collected from (98887..99420), so the
next regen cannot promote them: shell/leak (#36585: failed in the batch in
340 of 400 builds), request-clone-leak (excluded by hand in #36759 after
batch failures on x64-asan and Windows) and zlib/leak (build 86483). All
three are already in excludeFiles; this only makes the exclusion stick.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants