Skip to content

test: scale expect-extend intensive-usage iterations down on debug builds - #39359

Open
robobun wants to merge 2 commits into
mainfrom
farm/3a79cfd8/expect-extend-intensive-timeout
Open

robobun wants to merge 2 commits into
mainfrom
farm/3a79cfd8/expect-extend-intensive-timeout

Conversation

@robobun

@robobun robobun commented Aug 16, 2026 •

Copy link
Copy Markdown
Collaborator

Problem

  • bun bd test test/js/bun/test/expect-extend.test.js intermittently fails "should not crash under intensive usage" with this test timed out after 5000ms.
  • The test (expect-extend.test.js:331) runs 10,000 iterations of two custom-matcher expect() calls. On a debug build that loop takes 4.0s to 6.2s (9 runs of the file: timed out at 5493ms, 5659ms and 6166ms, two more passed at 4964ms and 4987ms), against bun test's 5s default per-test timeout (src/options_types/context.rs:483). The same loop takes 13ms to 23ms on a release build.
  • Only debug builds hit this. CI does not run a debug lane (.buildkite/ci.mjs), and its runner passes its own --timeout to every bun test invocation (90s, tripled to 270s on the ASAN lane; scripts/runner.node.mjs:976-1002 and :1969-1974), so the 5s default only applies when the file is run directly under a debug build, i.e. bun bd test. bun test: a matcher blocked on a promise that never settles fails with the test timeout #39089 ran into the same timeout while verifying an unrelated change on a debug build.

Fix

  • Run 1,000 iterations when isDebug, 10,000 otherwise (test/js/bun/test/expect-extend.test.js). Same shape as the other debug-sized loops in the tree (test/js/bun/yaml/yaml.test.ts:3121, test/js/web/html/FormData.test.ts:754), and the convention in test/CLAUDE.md is to shrink the work on slow builds rather than set a timeout.
  • The gate is isDebug only, not isDebug || isASAN: every build with the problem is a debug build, and the CI release ASAN lane has a 270s budget and has not been timing out, so it keeps the original 10,000 iterations. CI behavior is unchanged by this PR; only debug builds run the shorter loop.
  • What the test checks is unchanged: every iteration still builds an Expect object, calls the custom matcher through its wrapper, and runs the custom asymmetric matcher through toEqual; the Bun.gc(true) after the loop still collects everything the loop allocated; and the tests after it keep using the matchers registered at the top of the file, so a wrapper or matcher context freed by that collection would still fail the file at either count. With BUN_JSC_logGC=1 no collection starts inside the loop at either count (about 0.5KB allocated per iteration, under JSC's collection threshold), so the forced collection after the loop is the lifetime check at 10,000 iterations as well as at 1,000. The debug build is still ASAN-instrumented for each of the 1,000 iterations.
  • Verified with bun bd test test/js/bun/test/expect-extend.test.js: the test takes 468ms to 613ms on the debug build (13 of 13 timed runs across both revisions of this PR pass, all with BUN_GARBAGE_COLLECTOR_LEVEL=1 as CI sets it), down from 4.0s to 6.2s; the file reports 2088 expect() calls.
    • USE_SYSTEM_BUN=1 bun test test/js/bun/test/expect-extend.test.js (release) still reports 20088 expect() calls, i.e. the 10,000-iteration path.
  • Test-only change, so there is no runtime fix for a test to prove; the before and after timings above are the evidence. fix: expect.extend only iterates own properties #28365 and bun:test: populate customTesters, utils.diff, iterableEquality, and subsetEquality on the expect.extend matcher context #32936 each lower this count flat (for release builds too) as a side change of unrelated runtime changes; this PR fixes the timeout on its own and does not depend on either of them.

Background

  • isDebug (test/harness.ts:29) is Bun.version.includes("debug"): true for bun bd builds, false for release builds including the release ASAN build CI's ASAN lane runs.
  • Debug builds are slow on this loop partly because Rust debug_assertions are on: among other things every refcounted object created by the matcher path captures a stack trace in ThreadLock::lock (src/bun_core/util.rs:2142). The release ASAN profile also enables those assertions (scripts/build/rust.ts:742-750), but that lane is optimized code under a 270s per-test budget, so it does not need the shorter loop.
  • CI runs bun test with BUN_GARBAGE_COLLECTOR_LEVEL=1 (scripts/runner.node.mjs). In that mode every matcher call requests a collection on the way out (apply_custom_matcher and post_match in src/runtime/test_runner/expect.rs). withoutAggressiveGC, which already wraps this loop, turns that off for the duration of the loop, so the loop time above is the matcher work itself.
Timings (linux-x64, 16 cores, debug+ASAN build of 8326d1b)

Before, bun bd test test/js/bun/test/expect-extend.test.js, 9 runs:

(fail) should not crash under intensive usage [5659.50ms]   timed out
(fail) should not crash under intensive usage [6166.67ms]   timed out
(fail) should not crash under intensive usage [5493.14ms]   timed out
(pass) should not crash under intensive usage [4987.26ms]
(pass) should not crash under intensive usage [4964.46ms]
(pass) should not crash under intensive usage [4100.66ms]
(pass) should not crash under intensive usage [4139.27ms]
(pass) should not crash under intensive usage [4045.66ms]
(pass) should not crash under intensive usage [4205.88ms]

After (isDebug ? 1000 : 10000), same command, 5 runs with BUN_GARBAGE_COLLECTOR_LEVEL=1:

(pass) should not crash under intensive usage [473.24ms]
(pass) should not crash under intensive usage [477.79ms]
(pass) should not crash under intensive usage [478.56ms]
(pass) should not crash under intensive usage [501.28ms]
(pass) should not crash under intensive usage [468.03ms]

The first revision of this PR gated on isDebug || isASAN; on the same debug build it measured 470ms to 613ms over 8 runs (identical code path, since a debug build is both). It was changed to isDebug only so that the CI release ASAN lane, which has a 270s per-test budget and no observed timeout, keeps its 10,000 iterations. On that lane the first revision ran the whole file in 346ms (Buildkite build 99581).

Release (USE_SYSTEM_BUN=1, bun 1.4.0), 10,000 iterations, 4 runs: 13.51ms, 22.68ms, 22.82ms, 14.16ms.

GC activity inside the loop, standalone copy of the test with BUN_JSC_logGC=1 on the debug build: 0 collections started inside the loop at 1,000, 2,000 and 10,000 iterations (at 10,000 the only in-loop log lines are the end of an eden collection that started before the loop); in every case the Bun.gc(true) after the loop runs a FullCollection and full sweep. Loop time in that copy: 510ms at 1,000, 1194ms at 2,000, 4156ms at 10,000.


[stamp-90s] gate passed · iteration 0 · 1 files touched

passes on PR (with fix)
Test-only change.

Debug/ASAN (expected pass):
$ bun bd test 'test/js/bun/test/expect-extend.test.js'
$ BUN_DEBUG_QUIET_LOGS=1 bun scripts/build.ts --profile=debug --quiet test test/js/bun/test/expect-extend.test.js
bun test v1.4.0 (8326d1bd3)

test/js/bun/test/expect-extend.test.js:
(pass) is available globally when matcher is unary [11.05ms]
(pass) is available globally when matcher is variadic [6.90ms]
(pass) exposes matcherUtils in context [4.00ms]
(pass) is ok if there is no message specified [4.79ms]
(pass) works with empty matcher name [1.95ms]
(pass) exposes an equality function to custom matchers [4.54ms]
(pass) defines asymmetric unary matchers [8.78ms]
(pass) defines asymmetric unary matchers that can be prefixed by not [5.74ms]
(pass) defines asymmetric variadic matchers [5.41ms]
(pass) defines asymmetric variadic matchers that can be prefixed by not [5.93ms]
(pass) prints the Symbol into the error message [5.66ms]
(pass) allows overriding existing extension [4.49ms]
(pass) throws descriptive errors for invalid matchers [8.18ms]
(pass) invalid matcher implementations errors > handles correctly when matcher throws [4.04ms]
(pass) invalid matcher implementations errors > throws when returns undefined [5.72ms]
(pass) invalid matcher implementations errors > throws when returns not an object [3.88ms]
(pass) invalid matcher implementations errors > throws when return is missing "pass" [3.62ms]
(pass) invalid matcher implementations errors > supports undefined message [8.60ms]
(pass) invalid matcher implementations errors > handles correctly when "message" getter throws [5.04ms]
(pass) async support > supports async matcher result [9.14ms]
error: error
error
error: error
error
error: error
error
error: error
error
(pass) async support > throws on async matcher result rejection [25.66ms]
(pass) should not crash under intensive usage [463.61ms]
(pass) should propagate errors from calling .toString() on the message callback value [7.43ms]
(pass) should support asymmetric matchers [9.49ms]
(pass) works on prototypes [3.13ms]
(pass) works on classes
... (truncated)
Exit: 0
diff hotspot
test/js/bun/test/expect-extend.test.js | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

gate history · 1 passed · 0 rejected · iteration 0

evidence per changed file
file                                    reads  edits  tests
test/js/bun/test/expect-extend.test.js      1      4      0

…d ASAN builds

The "should not crash under intensive usage" test runs 10,000 iterations of
two custom-matcher expect() calls. On a debug+ASAN build that takes 4 to 6
seconds, so it intermittently hits the default 5 second per-test timeout.
Run 1,000 iterations there and keep 10,000 on release builds.
@coderabbitai

coderabbitai Bot commented Aug 16, 2026 •

Copy link
Copy Markdown
Contributor

Warning

Review limit reached

@robobun, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 9 minutes

Limit details: You’ve used all 5 included reviews currently available under your plan.

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro

Run ID: 681f0821-1ded-470a-9cf1-6f04256ca5f8

📥 Commits

Reviewing files that changed from the base of the PR and between 8326d1b and aa8b1a5.

📒 Files selected for processing (1)
  • test/js/bun/test/expect-extend.test.js

Comment @coderabbitai help to get the list of available commands.

@robobun

robobun commented Aug 16, 2026 •

Copy link
Copy Markdown
Collaborator Author
Updated 9:05 PM PT - Aug 16th, 2026

✅ @robobun, your commit aa8b1a5a2392bd66cb860a74fe078a4519a22548 passed in Build #99811! 🎉


🧪   To try this PR locally:

bunx bun-pr 39359

That installs a local version of the PR into your bun-39359 executable, so you can run:

bun-39359 --bun

@robobun

robobun commented Aug 16, 2026 •

Copy link
Copy Markdown
Collaborator Author

Status: test-only change, ready for review.

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM — test-only de-flake that shrinks the workload on instrumented builds instead of raising the timeout, matching repo convention.

What was reviewed:

  • isDebug and isASAN are both exported from test/harness.ts; the isDebug || isASAN ? small : large pattern is already used in ~10 other test files.
  • Release builds keep the full 10,000 iterations, so non-instrumented coverage is unchanged.
  • The property the test protects (no crash across many custom-matcher calls + forced full GC) is still asserted at 1,000 iterations under ASAN.
Extended reasoning...

Overview

This PR touches a single test file, test/js/bun/test/expect-extend.test.js, changing two lines: it adds isASAN and isDebug to the existing harness import, and replaces the hardcoded 10000 loop bound in the "should not crash under intensive usage" test with isDebug || isASAN ? 1000 : 10000. No runtime code, no other tests, no assertions are changed.

Security risks

None. This is a test-only iteration-count adjustment with no effect on shipped code, no new inputs, and no changes to any security-relevant path.

Level of scrutiny

Low. The change is mechanical and follows an established idiom — grep shows the identical isDebug || isASAN ? … : … scaling pattern in bundler_splitting.test.ts, html-rewriter.test.js, string-decoder.test.js, worker_heap_snapshot_gc.test.ts, and others. Both harness exports were verified to exist. The repo's own review guidance explicitly prefers shrinking workload over raising per-test timeouts, and this PR does exactly that.

Other factors

The test's intent is preserved: it is a crash/UAF stress test, and 1,000 iterations under ASAN (which is what surfaces UAF) plus the trailing Bun.gc(true) still exercises the same lifetime path the original 10,000 did — the PR description confirms via BUN_JSC_logGC=1 that no in-loop collection fires at either count, so the forced full collection after the loop is the load-bearing check in both cases. Release lanes retain the original 10,000 iterations. No prior reviewer comments are outstanding, and the bug-hunting system found nothing.

CI always passes its own --timeout (90s, 270s on the ASAN lane), so the
5s default only applies to local bun bd test runs, which are debug builds.
Keep the release ASAN lane at the original 10,000 iterations.
@robobun robobun changed the title test: scale expect-extend intensive-usage iterations down on debug and ASAN builds test: scale expect-extend intensive-usage iterations down on debug builds Aug 17, 2026
@robobun

robobun commented Aug 17, 2026

Copy link
Copy Markdown
Collaborator Author

Revised in aa8b1a5: the iteration count is now gated on isDebug alone instead of isDebug || isASAN.

The 5s default timeout that this test was hitting only applies to direct bun bd test runs: CI passes its own --timeout to every bun test invocation (scripts/runner.node.mjs:976-1002 and :1969-1974, 90s, tripled to 270s on the ASAN lane) and has no debug lane, so the ASAN half of the condition would only have reduced the release ASAN lane from 10,000 iterations to 1,000 without a timeout to fix there. With isDebug alone, CI keeps running the test exactly as it does today and only debug builds run the shorter loop. The review above predates this revision; the PR description has been updated to match.

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM — test-only iteration-count reduction on debug builds, matching the established pattern (e.g. sqlite-sql.test.ts:2237). The inline nit about the PR title/description wording is non-blocking.

Extended reasoning...

Overview

This PR touches a single test file, test/js/bun/test/expect-extend.test.js, changing the "should not crash under intensive usage" test to run 1,000 iterations on debug builds instead of 10,000 (release builds keep 10,000). The diff is 3 lines: import isDebug from harness, compute iterations, and use it as the loop bound. No runtime code is touched.

Security risks

None. Test-only change to an iteration count constant; no new inputs, no I/O, no external surface.

Level of scrutiny

Low. This is the canonical de-flaking pattern the repo guidelines explicitly prescribe ("shrink the workload" over raising timeouts), and the exact isDebug ? 1000 : 10000 shape already exists verbatim at test/js/sql/sqlite-sql.test.ts:2237. isDebug is exported from test/harness.ts:29. The test's invariant (custom matchers don't crash/UAF under repeated use, followed by a forced full GC) is preserved at 1,000 iterations — the PR description documents that no in-loop GC fires at either count, so the post-loop Bun.gc(true) is the lifetime check in both cases, and ASAN still instruments every iteration.

Other factors

The one finding is a nit: commit aa8b1a5 intentionally narrowed the gate from isDebug || isASAN to isDebug only, but the PR title/description still say "debug and ASAN". Release-ASAN at 10,000 iterations is well under the timeout (release is ~20ms; even 5× ASAN overhead is <200ms), so the code is correct as-is — only the description wording is stale. This does not block merge. No prior human reviews or outstanding comments to address.

Comment thread test/js/bun/test/expect-extend.test.js Resolved

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant