Skip to content

test(s3): make s3.leak.test.ts able to fail, and run it on the local S3 server - #46686

Open
robobun wants to merge 3 commits into
mainfrom
robobun/a7ee2488/s3-leak-test-local-server
Open

robobun wants to merge 3 commits into
mainfrom
robobun/a7ee2488/s3-leak-test-local-server

Conversation

@robobun

@robobun robobun commented Oct 24, 2026 •

Copy link
Copy Markdown
Collaborator

Problem

  • test/js/bun/s3/s3.leak.test.ts cannot fail for a leak. Each fixture calls its operation one time: new Array(100).fill(readLargeFile()) fills the array with one promise.
  • The file runs against Cloudflare R2 only. A request of 1 MiB to R2 sometimes takes longer than the limit of a case. Builds 130095, 130097 and 130714 show this test timed out after 30000ms.

Fix

  • One fixture, s3-leak-fixture.ts, replaces the five. It warms up with 30 operations, then measures 100, ten at a time. It fails if those 100 did not move 100 MiB.
  • The fixture runs against the local S3 server of test/packages/s3-server (test: replace the MinIO container with an S3 server on Bun.serve #44054), as s3.test.ts does. Nothing in this file needs R2.
  • The bound is 25 MiB (release) and 40 MiB (debug or ASAN). Without a leak the RSS grows by up to 6 and 10 MiB.
  • Verified: when each operation keeps its 1 MiB body, all five cases fail (103 to 118 MiB). bun bd test test/js/bun/s3/s3.leak.test.ts and the release build of main pass.

Background

  • A case runs the fixture in a child process. The fixture reads its RSS before and after the operations.
  • R2 gave this file one thing that the local server does not: TLS. The S3 client and fetch() share the HTTP client, and fetch-leak.test.ts has TLS cases.
  • Considered the loop fix alone (Fix s3 leak tests run wrong times. Related to #20487 #25526): it sends about 550 bodies of 1 MiB to R2 per attempt. Considered 100 operations at once: the peak memory then differs from the warm-up.
Notes

Each case fails with a leak and passes without it

The leak is in a scratch copy of the fixture: each operation keeps its body in an array (the chunks of stream(), the string of text(), a fresh Buffer for each write). The numbers are the growth in MiB that the test got.

build bound stream() text() writer() write() Bun.write without the leak
debug + ASAN, bun bd test 40 102.7 110.2 118.2 110.4 111.2 5 pass, 38 s
release of main, canary 7d240c1dd 25 106.6 103.4 106.6 103.4 103.0 5 pass, 3.4 s
release 367d939d9 (2026-09-18) 25 107.7 104.7 107.9 105.4 105.5 5 pass, 2.8 s

With the leak the result is 0 pass, 5 fail on each build, for example Expected: < 25, Received: 106.6015625.

The growth without a leak

build runs of the fixture growth, MiB
release of main, with the variables of the CI runner (BUN_GARBAGE_COLLECTOR_LEVEL=1, BUN_JSC_randomIntegrityAuditRate=1.0) 100 -0.1 to 4.3
release of main 100 -1.4 to 4.0
release 367d939d9 100 -1.8 to 6.2
debug + ASAN 20 1.2 to 9.8
debug + ASAN, with the variables of the CI runner on the ASAN lane 10 1.7 to 9.8
  • The growth is one step at about the 60th operation, when the optimizing JIT tier starts. With BUN_JSC_useDFGJIT=0 (release 367d939d9, write()) the RSS stays between 37.0 and 38.1 MiB from the second batch to the twelfth.
  • 300 operations grow a debug build by 5.5 to 11.8 MiB, and 900 operations grow the release build 367d939d9 by 2.2 to 6.1 MiB. So there is no growth per operation.
  • The old bound of 15 MB was never reached by a real run: the measured part was one operation.

The guard on the work

The fixture adds up the bytes that the operations moved, and throws if the sum is not 100 MiB. The sum is made in the function that Array.from calls for each element.

  • With the loop of the old fixtures (new Array(10).fill(promise)) each case fails: error: The 100 operations moved 10485760 bytes, not 100 MiB.
  • With an empty object to download, the two read cases fail: error: The 100 operations moved 0 bytes, not 100 MiB.

The reading of the RSS

  • After Bun.gc(true) the RSS falls for about 200 ms more, in steps about 100 ms apart. In a sample from the release build of 09-18 it went from 55.7 to 44.8 MiB.
  • The old fixtures read 10 ms after the collection. With that reading the growth without a leak is -1.4 to 7.4 MiB on main and -4.8 to 20.2 MiB on the build of 09-18.
  • The fixture reads each 20 ms until ten readings in a row are not below the lowest one. It stops after 250 readings at the latest.

ASAN

  • With the quarantine of ASAN, a fixture without a leak grows by 141 to 208 MiB: each freed body stays in the quarantine. expectRssDeltaBelow turns the quarantine off. It now takes the environment of the fixture as a third argument, with bunEnv as the default. No other caller changes.
  • test/no-validate-leaksan.txt still lists the file, so LeakSanitizer stays off for it.

The environment of the fixture

  • Bun.s3 reads S3_* before AWS_* (src/dotenv/env_loader.rs). The old test set AWS_*, so an S3_BUCKET of the machine won: NoSuchBucket. The test sets S3_* now.
  • A session token of the machine makes the local server answer InvalidToken. The test clears both names.
  • For an http endpoint the S3 client and fetch() follow HTTP_PROXY, http_proxy, ALL_PROXY and all_proxy, also for a loopback address. With one of them set, the fixture gets ConnectionRefused and the server does not start (The s3-server process ended before it listened). The test clears the four names for both. HTTPS_PROXY is not read for an http endpoint.
  • With all of these set on the machine, the file passes.

One at a time on a debug or ASAN build

The fixtures share one server process. A debug build of the server needs 13 to 27 ms for a request.

bun bd test, this file each case the file
five fixtures at the same time 9 to 24 s 22 to 32 s
one at a time 5 to 11 s 38 to 46 s
  • A debug build gives a case 30 s, and one at a time keeps each case far from that limit. Other builds keep the default limit: 5 s for a local bun test, 90 s from the CI runner, 270 s on the ASAN lane.
  • A release build runs the five at the same time: about 3 s for the file. One at a time it takes 8 s.
  • A server for each fixture does not help on a debug build: 11 to 17 s for each case.

What changed in the operations

  • The body is 1 MiB of A. It was 1 MiB of zero bytes, and 1 MiB + 1 byte in s3-write-leak-fixture.js. The client has no limit at 1 MiB: a part is 5 MiB.
  • The object that the read cases download is not deleted at the end. The server keeps its data in memory and stops with the test.
  • The copy of the fixture to a temporary directory and its out.bin argument are gone. No fixture read them.
  • The calls to stream(), text(), writer(), write(), Bun.write and unlink() are the same.

What the file sends to R2 in one attempt

  • Before: 12 bodies of 1 MiB.
  • With the loop fix alone: 552 bodies of 1 MiB.
  • Now: nothing.

test/flaky-tests.txt

The entry of the file (1 of 80 builds; windows) is removed. The three builds above show the cause: the 30 s limit on a request to R2. A flake of the new file is then reported as a new flake.

CI results

The bounds come from measurements on Linux x64. CI ran the file on the other platforms for the first time. Each cell is the time of the whole file, with 5 pass, 0 fail.

lane build 131667 build 131712 build 131737
debian 13 x64 1.97 s 2.14 s 1.97 s
debian 13 aarch64 1.97 s 1.82 s 1.98 s
ubuntu 25.04 x64 2.02 s 1.73 s 1.86 s
ubuntu 25.04 aarch64 1.95 s 1.97 s 1.89 s
alpine 3.23 x64 2.01 s 1.84 s 1.66 s
alpine 3.23 aarch64 1.84 s 2.02 s 1.83 s
windows 2019 x64 2.25 s 2.22 s 2.02 s
windows 11 aarch64 2.05 s 2.34 s 2.17 s
macOS aarch64 2.16 s 5.14 s 4.65 s
debian 13 x64 with ASAN 6.15 s 6.50 s 7.38 s

Build 131737 has the limit of the last commit: only a debug build sets one. macOS x64 has not run the file yet: its test jobs got no agent in any of the three builds.

Other pull requests


[auto-merge] gate passed · iteration 2 · 9 files touched

passes on PR (with fix)
Test-only change.

Debug/ASAN (expected pass):
$ bun bd test 'test/js/bun/s3/s3.leak.test.ts'
$ BUN_DEBUG_QUIET_LOGS=1 bun scripts/build.ts --profile=debug --quiet test "test/js/bun/s3/s3.leak.test.ts"
bun test v1.4.4 (367d939d9)

test/js/bun/s3/s3.leak.test.ts:
(pass) s3 > leak tests > s3().stream() should not leak [13670.52ms]
(pass) s3 > leak tests > s3().text() should not leak [7242.21ms]
(pass) s3 > leak tests > s3().writer().write() should not leak [10246.53ms]
(pass) s3 > leak tests > s3().write() should not leak [6368.18ms]
(pass) s3 > leak tests > Bun.write should not leak [6694.60ms]

 5 pass
 0 fail
 15 expect() calls
Ran 5 tests across 1 file. [68.63s]
Exit: 0
diff hotspot
test/flaky-tests.txt                     |   1 -
 test/harness.ts                          |   6 +-
 test/js/bun/s3/bun-write-leak-fixture.js |  32 -----
 test/js/bun/s3/s3-leak-fixture.ts        | 101 +++++++++++++++
 test/js/bun/s3/s3-stream-leak-fixture.js |  41 ------
 test/js/bun/s3/s3-text-leak-fixture.js   |  36 ------
 test/js/bun/s3/s3-write-leak-fixture.js  |  32 -----
 test/js/bun/s3/s3-writer-leak-fixture.js |  40 ------
 test/js/bun/s3/s3.leak.test.ts           | 212 ++++++++-----------------------
 9 files changed, 155 insertions(+), 346 deletions(-)

gate history · 1 passed · 0 rejected · iteration 2

evidence per changed file
file                                      reads  edits  tests
test/flaky-tests.txt                          0      0     50
test/harness.ts                               1      1     55
test/js/bun/s3/bun-write-leak-fixture.js      0      0     45
test/js/bun/s3/s3-leak-fixture.ts             2      4     52
test/js/bun/s3/s3-stream-leak-fixture.js      0      0     45
test/js/bun/s3/s3-text-leak-fixture.js        0      0     45
test/js/bun/s3/s3-write-leak-fixture.js       0      0     45
test/js/bun/s3/s3-writer-leak-fixture.js      0      0     45
test/js/bun/s3/s3.leak.test.ts                2      3     45

…S3 server

Each of the five leak fixtures called its operation one time:
`new Array(100).fill(fn())` fills the array with one promise. A case compared
the RSS after one operation with the RSS after one operation, so a leak could
not make it fail. The file also ran against Cloudflare R2 only, where one
request of 1 MiB sometimes takes longer than the 30 s limit of a case.

- One fixture, s3-leak-fixture.ts, replaces the five. It does 30 operations
  as a warm-up and then measures 100, ten at a time. It fails if the 100
  operations did not move 100 MiB.
- The fixture runs against the server of test/packages/s3-server. Nothing in
  the file needs R2.
- The fixture reads the RSS until it stops falling. The test uses
  expectRssDeltaBelow, which now takes the environment of the fixture.
- The entry of the file in test/flaky-tests.txt goes away: the flake was the
  time limit on R2.

Co-authored-by: Jax <jaxyang39@gmail.com>
@robobun

robobun commented Oct 24, 2026 •

Copy link
Copy Markdown
Collaborator Author

Status: the diff is green. CI cannot finish, because the macOS x64 lane has no agent. The next step is a review by a maintainer.

CI, build 131737 (head 58b0d75):

  • 179 of 181 jobs passed: each lane on Linux, Windows and macOS aarch64, and the ASAN lane.
  • The two darwin x64 - test-bun jobs did not start. No agent took them, and they expired after 82 minutes. That lane also got no agent in builds 131667 and 131712.
  • test/js/bun/s3/s3.leak.test.ts passed on each of the ten lanes that ran it, in three builds (5 pass, 0 fail). The file takes about 2 s on Linux and Windows, 2 to 5 s on macOS aarch64, and 6 to 7.5 s on the ASAN lane.
  • Three other files failed one time and passed on a retry: spawn_waiter_thread.test.ts, node-http2.test.js, nth-anplusb-ident.test.ts. This PR does not touch them.

How I reproduced the problem:

  • On main, git grep -n "new Array(100).fill" -- test/js/bun/s3 shows the loop in each of the five fixtures. The array gets one promise, so a fixture does one operation where it says 100.
  • The file is skipped on a machine without the S3_R2_* secrets. In builds 130095, 130097 and 130714 a case of this file failed with this test timed out after 30000ms.

How I checked the new file:

  • bun bd test test/js/bun/s3/s3.leak.test.ts (debug build with ASAN) and the release build of main pass.
  • With a scratch copy of the fixture in which each operation keeps its body, all five cases fail on both builds. The table is in the Notes of the description.

@robobun

robobun commented Oct 24, 2026 •

Copy link
Copy Markdown
Collaborator Author
Updated 2:55 AM PT - Oct 24th, 2026

❌ @robobun, your commit 58b0d75 has 3 failures in Build #131737 (All Failures):


🧪   To try this PR locally:

bunx bun-pr 46686

That installs a local version of the PR into your bun-46686 executable, so you can run:

bun-46686 --bun

@coderabbitai

coderabbitai Bot commented Oct 24, 2026 •

Copy link
Copy Markdown
Contributor

Review in Change Stack →

No actionable comments were generated in the recent review.

Recent review info
Run configuration
Commits

Reviewing files that changed from the base of the PR and between a8b5e92 and 95f85d6.

Files selected for processing: 9
  • test/flaky-tests.txt
  • test/harness.ts
  • test/js/bun/s3/bun-write-leak-fixture.js
  • test/js/bun/s3/s3-leak-fixture.ts
  • test/js/bun/s3/s3-stream-leak-fixture.js
  • test/js/bun/s3/s3-text-leak-fixture.js
  • test/js/bun/s3/s3-write-leak-fixture.js
  • test/js/bun/s3/s3-writer-leak-fixture.js
  • test/js/bun/s3/s3.leak.test.ts
Files with no reviewable changes: 6
  • test/flaky-tests.txt
  • test/js/bun/s3/bun-write-leak-fixture.js
  • test/js/bun/s3/s3-write-leak-fixture.js
  • test/js/bun/s3/s3-stream-leak-fixture.js
  • test/js/bun/s3/s3-text-leak-fixture.js
  • test/js/bun/s3/s3-writer-leak-fixture.js

Included review availability: This review used your included allowance. Your plan provides up to 10 included reviews per hour; 2 remain after this review.

Walkthrough

The S3 leak tests now use a local S3 server and a shared TypeScript fixture to measure RSS across five operations. The fixture batches operations, settles RSS readings, checks transferred bytes, and reports the delta. The harness accepts a custom child-process environment, and the previous operation-specific fixtures were removed.

Changes

S3 leak test flow

Layer / File(s) Summary
Shared fixture operations
test/js/bun/s3/s3-leak-fixture.ts, test/js/bun/s3/*-leak-fixture.js
The shared fixture defines five selectable S3 operations and runs them in batches. The previous operation-specific fixtures were removed.
RSS measurement
test/js/bun/s3/s3-leak-fixture.ts
The fixture warms up, settles RSS readings, measures ten batches, checks that the operations moved 100 MiB, and prints the RSS delta.
Local server and test integration
test/js/bun/s3/s3.leak.test.ts, test/harness.ts, test/flaky-tests.txt
The tests configure a local S3 server and run the shared fixture with per-operation RSS limits. The harness accepts a custom child-process environment, and the flaky-test entry is removed.

Possibly related PRs

  • oven-sh/bun#46605: Moves large-transfer cases to a local S3 server instead of relying on Cloudflare R2.
  • oven-sh/bun#46581: Also uses RSS-based subprocess leak checks and addresses ASAN quarantine handling.
  • oven-sh/bun#46653: Changes S3 blob size resolution, which can affect the read operations used by this fixture.

Suggested reviewers: jarred-sumner

Priority: Normal

Merge Risk: Minimal · up to 95f85

The consolidated leak tests have no identified merge-blocking issue; proceed with normal test checks.

Pre-merge checks: Passed 4
Check name Status Explanation
Title check Passed The title clearly identifies both primary changes: making the S3 leak test effective and running it against the local S3 server.
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Description check Passed The description is complete and relevant. It explains the problem, implementation, verification steps, test results, scope, and background. It does not use the exact template headings, but it provides…
Full details: Description check

Explanation

The description is complete and relevant. It explains the problem, implementation, verification steps, test results, scope, and background. It does not use the exact template headings, but it provides the required information in equivalent sections.

Finishing Touches: 💡 1
🛠️ Fix failing CI checks: 💡
  • Commit to this branch
  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nothing blocking. The comments below are optional suggestions. There is no need to push a fix for them before merging.

Beyond the inline finding, I also checked: S3_* is read ahead of AWS_* in src/dotenv/env_loader.rs, so the fixture's env wins over machine credentials; the top-level spawnServer plus afterAll(stop) shape and the region-bearing clientOptions("buntest") match what s3.test.ts already does; the new defaulted env parameter on expectRssDeltaBelow leaves every other caller unchanged; and writer.end() is typed as the byte count, with the fixture's moved !== 100 * MiB guard throwing rather than passing vacuously if that ever changed.

Extended reasoning...

Test-only change: one consolidated S3 leak fixture replaces five, the test now runs against the in-tree s3-server instead of R2, and the harness helper gains a backward-compatible env parameter. No runtime source or security-sensitive surface is touched. The test previously skipped on every lane without R2 secrets and now asserts RSS thresholds on all CI platforms for the first time, which is why a human look at the first CI run is still worthwhile.

Comment thread test/js/bun/s3/s3.leak.test.ts Resolved
The fixed limit of 30 s was below the limit that the CI runner sets for a
test: 90 s, or 270 s on the ASAN lane. Only a debug build needs more than the
default limit of `bun test` (5 s): a case takes up to 11 s there.

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I reviewed the updated PR (including the follow-up that limits the 30 s per-test timeout to debug builds) and found no bugs; since the RSS bounds and the per-case timing were measured on Linux x64 only, a human look at the first CI results on the other lanes is still worthwhile.

What was reviewed:

  • The claim that S3_* wins over AWS_* is true: src/dotenv/env_loader.rs get_s3_credentials reads S3_* first for all six names, including the session token the test unsets.
  • spawnServer passes the given env straight to Bun.spawn, so the undefined proxy entries drop those variables; the top-level await spawnServer + afterAll(stop) shape and the buntest bucket match s3.test.ts.
  • it.concurrentIf exists in bun:test types; expectRssDeltaBelow still applies withoutAsanQuarantine to the new env argument, and the file remains listed in test/no-validate-leaksan.txt.
  • The fixture's 100 MiB "moved" guard would fail loudly rather than silently if writer.end() or a read returned fewer bytes, and the settle loop is bounded (250 readings).
Extended reasoning...

Test-only change: it rewrites test/js/bun/s3/s3.leak.test.ts to spawn the in-repo s3-server instead of using R2 secrets, replaces five JS fixtures with one parameterized TypeScript fixture that measures settled RSS over 100 one-MiB operations, adds an optional env parameter to expectRssDeltaBelow in test/harness.ts, and removes the file's flaky-tests.txt entry. No production code or security-sensitive surface is touched. The structure mirrors the existing s3.test.ts usage of spawnServer, and the env-precedence and env-passthrough claims check out against src and the s3-server package. Deferring rather than approving because the thresholds and the debug-only 30 s cap (still below the CI runner's 90 s / 270 s ceiling) rest on measurements from one platform, and the PR itself notes CI is the first run elsewhere.

@robobun

robobun commented Oct 24, 2026

Copy link
Copy Markdown
Collaborator Author

The first CI results on the other lanes are in the description now (Notes, "CI results").

In builds 131667 and 131712 the file passed on ten lanes, each time with 5 pass, 0 fail:

  • Linux (glibc and musl, x64 and aarch64) and Windows (x64 and aarch64): 1.73 to 2.34 s for the file.
  • macOS aarch64: 2.16 s and 5.14 s.
  • The ASAN lane: 6.15 s and 6.50 s.

macOS x64 has not run the file yet. Its test jobs got no agent in either build.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant