Repository navigation
Conversation
…rocess stop() was calling this.process.kill(0). In the Node child_process API, signal 0 is a liveness probe: it checks whether the process exists and returns true/false, but sends no signal. The forked verdaccio process therefore kept running after every test file's afterAll(), leaking one process per file that uses VerdaccioRegistry. This restores the default SIGTERM (the behavior before #16540) and suppresses the exit listener's 'exited with signal' log when the exit was triggered by stop(), so normal teardown stays quiet. A new test starts a registry, calls stop(), and asserts the child's exit event fires.
Contributor
|
Warning Review limit reached
Next review available in: 8 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: ASSERTIVE Plan: Pro Run ID: 📒 Files selected for processing (2)
Comment |
Collaborator
Author
|
Reproduced by starting a CI status: the diff is green. The new test passed on every lane in both build 85129 and build 85143. Remaining red on 85143 is 11 tests all tagged flaky by the annotation tooling (napi threadsafe finalizer, spawn-streaming-stdout, svelte client-side, grpc-js subchannel pool, etc.), each of which passed alone or on retry and none of which import Self-review raised no surviving concerns; the |
Contributor
|
This PR may be a duplicate of:
🤖 Generated with Claude Code |
Collaborator
Author
|
Re: duplicate check pointing at #34687. That PR does include the same |
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
LGTM — test-harness-only fix; kill(0) → kill() is the correct restoration of pre-#16540 behavior. The inline nit about resetting #stopped in start() is non-blocking (no current caller restarts a registry, and the only casualty would be a suppressed diagnostic log).
Extended reasoning...
Overview
This PR touches two files, both under test/:
test/harness.ts: changesVerdaccioRegistry.stop()fromthis.process?.kill(0)tothis.process?.kill(), and adds a#stoppedflag so theexitlistener stays quiet during intentional teardown while still logging unexpected mid-test crashes.test/cli/install/verdaccio-registry-lifecycle.test.ts: new test that starts a registry, callsstop(), and asserts the child'sexitevent fires (viaonce(child, "exit")raced against a bounded 3s sleep), with anafterAllSIGKILL fallback for cleanup.
No runtime (src/) code is touched.
Security risks
None. This is test infrastructure; the change sends SIGTERM to a test-spawned child process instead of a no-op signal-0 probe.
Level of scrutiny
Low. The core fix is a one-token change whose correctness follows directly from Node's documented child_process.kill() semantics — signal 0 only tests process existence and never delivers a signal. The #stopped flag is a small quality-of-life addition whose only effect is suppressing a console.error during normal teardown. The new test follows harness conventions: awaits the actual observable condition (exit event) rather than sleeping, uses a bounded race for the timeout branch, and cleans up with SIGKILL in afterAll if the assertion fails.
Other factors
The bug-hunting system flagged one nit: #stopped is set in stop() but never reset in start(), so a start→stop→start sequence would carry the stale flag into the second lifecycle. I confirmed via the finding's own analysis that no current test/ consumer restarts a VerdaccioRegistry instance, and the worst case is a suppressed diagnostic log — no test would pass or fail incorrectly. This is worth a one-line follow-up but does not block the fix, which resolves a real process leak affecting every install test file today.
Keeps the flag's invariant correct if an instance is ever restarted.
Collaborator
Author
|
Updated 9:31 AM PT - Jul 29th, 2026
✅ @robobun, your commit d652713ce2dba55159662963c02e7996149669ab passed in 🧪 To try this PR locally: bunx bun-pr 36352That installs a local version of the PR into your bun-36352 --bun |
Contributor
There was a problem hiding this comment.
LGTM — the kill(0) → kill() fix is unambiguously correct and the change is entirely test-harness code. The remaining inline nit is about a spurious console.error in a restart sequence no current test exercises; it doesn't affect the fix and can be a follow-up.
Extended reasoning...
Overview
This PR fixes VerdaccioRegistry.stop() in test/harness.ts, which was calling this.process?.kill(0) — a liveness probe that sends no signal — instead of kill(), so the forked verdaccio child leaked past every afterAll(). The fix restores the default SIGTERM, adds a #stopped flag to suppress the resulting exit-log noise on intentional teardown, and adds test/cli/install/verdaccio-registry-lifecycle.test.ts which awaits the child's exit event after stop() and fails on the old code.
Security risks
None. Test-harness only; no runtime, network, auth, or crypto surface touched.
Level of scrutiny
Low. Both changed files live under test/. The functional change is a single well-understood Node child_process semantics fix (kill(0) vs kill()), and the rest is diagnostic-log plumbing plus a regression test. Nothing here affects the shipped binary.
Other factors
- My previous review's nit (reset
#stoppedinstart()) was addressed in e9a722c. This run's inline comment points out that the reset now races the previous child's asyncexitevent in a tightstop(); await start()sequence, so child A's SIGTERM would still log. That is real but nit-level: the only casualty is aconsole.errorin test-harness code, and a grep confirms every current consumer uses a singlebeforeAll(start)/afterAll(stop)lifecycle with no restart. It does not affect the actual process-leak fix. - The new test awaits the actual observable condition (
once(child, 'exit')) raced against a 3s bound rather than sleep-then-check, and has anafterAllSIGKILL safety net so a failure doesn't itself leak a process. - The duplicate-PR bot flagged #34687, which contains the same one-liner inside a much larger refactor; the author acknowledged and this focused fix can land independently.
Contributor
There was a problem hiding this comment.
LGTM — the kill(0) → kill() fix is correct and the #stopped flag handles the one lifecycle pattern in actual use. The remaining restart-ordering nit (resolved above) is out of scope for this focused fix and only affects a console.error in a codepath nobody exercises.
What was reviewed:
kill()semantics — argless defaults to SIGTERM, restoring pre-#16540 behavior- New test awaits the actual
exitevent with a bounded race and has anafterAllSIGKILL guard so a failure can't leak the child #stoppedflag lifecycle acrossstart()/stop()for the single-use pattern all ~15 consumers follow
Extended reasoning...
Overview
Test-harness-only change: VerdaccioRegistry.stop() in test/harness.ts was calling this.process?.kill(0), which in Node's child_process API only probes liveness without sending a signal. The fix drops the argument so the default SIGTERM is sent. A private #stopped flag suppresses the exit listener's diagnostic log during intentional teardown, and start() resets it. A new test file starts a registry, calls stop(), and asserts the child's exit event fires within 3s.
Security risks
None. Entirely under test/; no production code, no network exposure, no auth/crypto paths.
Level of scrutiny
Low. This is test infrastructure with a well-understood one-line semantic fix (kill(0) vs kill() is textbook Node behavior). The supporting #stopped flag only gates a console.error — worst-case failure mode is a spurious or missing log line, not incorrect test results.
Other factors
I left two inline nits on earlier revisions of this PR, both now resolved: the first (#stopped never reset) was fixed in e9a722c; the second (reset-vs-async-exit ordering on restart) was reasonably declined as out of scope since no consumer restarts a registry instance and the only effect is one extra log line. The new test follows the harness conventions — awaits the event rather than sleeping, uses a bounded Promise.race deadline, and cleans up with SIGKILL in afterAll so a failing assertion can't leak the process. The duplicate-PR check flagged #34687, which the author addressed: that PR bundles the same one-liner inside a larger refactor without a covering test, so this focused fix can land independently.
Collaborator
Author
|
Extra weight for this fix: the leaked verdaccio processes are actively breaking an unrelated test on darwin. One more hardening worth considering while in this code: |
Jarred-Sumner
pushed a commit
that referenced
this pull request
Aug 9, 2026
`test/js/bun/http/bun-server.test.ts` has been flaking on darwin lanes since the graceful `stop()` drain work landed (#35130 on Aug 5, #37074 on Aug 6): about 30 failing builds in the 36 hours after the second merge, plus the 90s-timeout reported red in build [90105](https://buildkite.com/bun/bun/builds/90105). Three distinct failure modes, each with a verified cause. This PR is test-only; the code paths the tests pin are unchanged and still covered (see Verification). ## 1. GC churn test answered by a foreign server `request on a connection surviving graceful stop() never reaches a collected handler` failed with ``` FAIL fetch round 11: bad initial response [{"status":200,"body":"\n <!DOCTYPE html>\...//x/-/static/main.55cdd4aea43adabcb109.js\"></script>..."}] ``` That body is a verdaccio web UI page; the `routes` variant got verdaccio's 404 `{"error":"no such package available"}`. A freshly dialed connection to the fixture's own freshly bound port was answered by an npm test registry. Cause, reproduced on a macOS 14 CI host: the ephemeral port allocator on macOS honors only exact-address conflicts. A wildcard `Bun.serve({ port: 0 })` bind gets handed a port that another process already holds at `127.0.0.1` (one collision per ~16,384 binds, i.e. once per wrap of the ephemeral range; a loopback-bound holder under both a real node and a bun process, same result). Connects to `127.0.0.1:port` then reach the more specific foreign listener. The churn fixture performs hundreds of wildcard binds per run, so it finds such a port regularly, and leaked verdaccio processes supply the listeners: `VerdaccioRegistry.stop()` calls `kill(0)`, which is a liveness probe rather than a kill (regressed in #16540; fix open in #36352). Fix: bind the fixture's servers (and decoys) to `127.0.0.1`, the address the parks dial. Verified on the same host: 36,000 loopback-bound port-0 binds against a live loopback holder, zero collisions. This protects the test regardless of what else leaks on the machine; #36352 independently removes the main leak source. ## 2. Parse-window test 90s timeout (the build 90105 red) ``` ✗ server.stop() drain promise counts open connections > a response completing inside another socket's parse window still closes its drained connection [90001.99ms] ^ this test timed out after 90000ms. ``` The fixture wrote a partial request head on connection A and waited 20 `setImmediate` ticks before `stop(false)`. A partial head gives the server nothing observable, and on a loaded darwin host those ticks can elapse before the server has even accepted the socket. Verified on a macOS 14 CI host: when `stop(false)` runs while a handshake-completed connection is still waiting in the accept queue, macOS strands it - never accepted, never counted, never closed, and the client side stays silently open. (Linux keeps the pending accept, which is why this never fired there.) In the fixture that means `/poke` never dispatches, `releaseB` never resolves, B stays parked, and `while (!a.closed || !b.closed)` spins until the test timeout with no other output - exactly the observed signature. Fix: A's poke is now a `POST` with a held body. The handler's dispatch is awaited before `stop()` (so the sweep provably sees a busy connection), and writing the 2-byte body afterwards completes the request. The body's fin chunk is delivered inside A's parse window, so B's completion still runs in that window's microtask drain - the per-socket close-gate property the test exists to pin is exercised exactly as before. ## 3. Mid-request sparing test resolving early Build [89946](https://buildkite.com/bun/bun/builds/89946) hit `error: stop() resolved while a mid-request connection was open` in `a connection mid-request survives stop() until the client closes`. Same root cause: the connection's partial head had not reached the server when `stop()` ran, so the server (correctly) had nothing to count and the drain promise resolved. Fix: the mid-request state is staged as a partial second head on a keep-alive connection that already completed a full request, so the server demonstrably owns the socket. If the sweep still closes it, the head had not arrived and the connection was legitimately idle - that round proves nothing about sparing, so it is voided and retried on a fresh server (bounded, fails loudly if every round races). A resolution while the socket is left open is still reported as the bug it would be. Confirmed-spared rounds run in two variants, and the test requires one success of each: - destroy: the client hangs up and that resolves the drain (the original assertion). - complete: the client finishes the head after `stop()`. The request must still dispatch and be answered, which requires the close-when-idle mark to survive the dispatch's response-state reset (`HTTP_CONNECTION_SCOPED` in uWS `HttpResponseData.h`), and the mark must then close the served connection. This preserves the dispatch-after-stop coverage the old parse-window staging provided incidentally (its partial head completed after `stop()`), which the redesign in section 2 otherwise moves ahead of the stop. ## Verification - `bun bd test test/js/bun/http/bun-server.test.ts`: 74 pass / 3 fail, the 3 failures (`parse source map and fetch small stream`, `rejected promise handled by error method`, `abrubtly close a upload request`) fail identically on an unmodified checkout in this environment. - Drain block run 5x, churn test 2x locally: all pass. - Both redesigned fixtures extracted and run 30x on a macOS 14 CI host under the exact canary from build 90105 (`89d30ad11`): 60/60 pass, plus 40x under a 14-way CPU-spin load: all pass. - Coverage check for the parse-window test: patching the `internalEnd` close gates back to the context-wide `isParsingHttp` bit (the bug #37074 guards against) makes the redesigned test fail by timeout, and restoring the per-socket gate makes it pass. - Coverage check for the mid-request test's complete variant: dropping `HTTP_CLOSE_WHEN_IDLE` from `HTTP_CONNECTION_SCOPED` (so the mark is wiped when the post-stop request dispatches) makes it fail by timeout; restored, it passes. The updated fixture also runs 30/30 on the macOS 14 CI host under the build 90105 canary, and hangs as expected under a pre-#37074 build. The remaining hazard - any long-running wildcard port-0 listener on darwin CI can have its loopback traffic stolen by a later explicit `127.0.0.1` bind such as `VerdaccioRegistry`'s `randomPort()` (range 1024-65535 overlaps the kernel's ephemeral range) - is worth addressing in the harness separately; noted on #36352. <!-- robobun:evidence:begin --> --- **no test proof** · iteration 0 · Platform-specific test-only change; deferring to CI. <!-- robobun:evidence:end -->
springmin
pushed a commit
to springmin/bun
that referenced
this pull request
Aug 10, 2026
…37141) `test/js/bun/http/bun-server.test.ts` has been flaking on darwin lanes since the graceful `stop()` drain work landed (oven-sh#35130 on Aug 5, oven-sh#37074 on Aug 6): about 30 failing builds in the 36 hours after the second merge, plus the 90s-timeout reported red in build [90105](https://buildkite.com/bun/bun/builds/90105). Three distinct failure modes, each with a verified cause. This PR is test-only; the code paths the tests pin are unchanged and still covered (see Verification). ## 1. GC churn test answered by a foreign server `request on a connection surviving graceful stop() never reaches a collected handler` failed with ``` FAIL fetch round 11: bad initial response [{"status":200,"body":"\n <!DOCTYPE html>\...//x/-/static/main.55cdd4aea43adabcb109.js\"></script>..."}] ``` That body is a verdaccio web UI page; the `routes` variant got verdaccio's 404 `{"error":"no such package available"}`. A freshly dialed connection to the fixture's own freshly bound port was answered by an npm test registry. Cause, reproduced on a macOS 14 CI host: the ephemeral port allocator on macOS honors only exact-address conflicts. A wildcard `Bun.serve({ port: 0 })` bind gets handed a port that another process already holds at `127.0.0.1` (one collision per ~16,384 binds, i.e. once per wrap of the ephemeral range; a loopback-bound holder under both a real node and a bun process, same result). Connects to `127.0.0.1:port` then reach the more specific foreign listener. The churn fixture performs hundreds of wildcard binds per run, so it finds such a port regularly, and leaked verdaccio processes supply the listeners: `VerdaccioRegistry.stop()` calls `kill(0)`, which is a liveness probe rather than a kill (regressed in oven-sh#16540; fix open in oven-sh#36352). Fix: bind the fixture's servers (and decoys) to `127.0.0.1`, the address the parks dial. Verified on the same host: 36,000 loopback-bound port-0 binds against a live loopback holder, zero collisions. This protects the test regardless of what else leaks on the machine; oven-sh#36352 independently removes the main leak source. ## 2. Parse-window test 90s timeout (the build 90105 red) ``` ✗ server.stop() drain promise counts open connections > a response completing inside another socket's parse window still closes its drained connection [90001.99ms] ^ this test timed out after 90000ms. ``` The fixture wrote a partial request head on connection A and waited 20 `setImmediate` ticks before `stop(false)`. A partial head gives the server nothing observable, and on a loaded darwin host those ticks can elapse before the server has even accepted the socket. Verified on a macOS 14 CI host: when `stop(false)` runs while a handshake-completed connection is still waiting in the accept queue, macOS strands it - never accepted, never counted, never closed, and the client side stays silently open. (Linux keeps the pending accept, which is why this never fired there.) In the fixture that means `/poke` never dispatches, `releaseB` never resolves, B stays parked, and `while (!a.closed || !b.closed)` spins until the test timeout with no other output - exactly the observed signature. Fix: A's poke is now a `POST` with a held body. The handler's dispatch is awaited before `stop()` (so the sweep provably sees a busy connection), and writing the 2-byte body afterwards completes the request. The body's fin chunk is delivered inside A's parse window, so B's completion still runs in that window's microtask drain - the per-socket close-gate property the test exists to pin is exercised exactly as before. ## 3. Mid-request sparing test resolving early Build [89946](https://buildkite.com/bun/bun/builds/89946) hit `error: stop() resolved while a mid-request connection was open` in `a connection mid-request survives stop() until the client closes`. Same root cause: the connection's partial head had not reached the server when `stop()` ran, so the server (correctly) had nothing to count and the drain promise resolved. Fix: the mid-request state is staged as a partial second head on a keep-alive connection that already completed a full request, so the server demonstrably owns the socket. If the sweep still closes it, the head had not arrived and the connection was legitimately idle - that round proves nothing about sparing, so it is voided and retried on a fresh server (bounded, fails loudly if every round races). A resolution while the socket is left open is still reported as the bug it would be. Confirmed-spared rounds run in two variants, and the test requires one success of each: - destroy: the client hangs up and that resolves the drain (the original assertion). - complete: the client finishes the head after `stop()`. The request must still dispatch and be answered, which requires the close-when-idle mark to survive the dispatch's response-state reset (`HTTP_CONNECTION_SCOPED` in uWS `HttpResponseData.h`), and the mark must then close the served connection. This preserves the dispatch-after-stop coverage the old parse-window staging provided incidentally (its partial head completed after `stop()`), which the redesign in section 2 otherwise moves ahead of the stop. ## Verification - `bun bd test test/js/bun/http/bun-server.test.ts`: 74 pass / 3 fail, the 3 failures (`parse source map and fetch small stream`, `rejected promise handled by error method`, `abrubtly close a upload request`) fail identically on an unmodified checkout in this environment. - Drain block run 5x, churn test 2x locally: all pass. - Both redesigned fixtures extracted and run 30x on a macOS 14 CI host under the exact canary from build 90105 (`89d30ad11`): 60/60 pass, plus 40x under a 14-way CPU-spin load: all pass. - Coverage check for the parse-window test: patching the `internalEnd` close gates back to the context-wide `isParsingHttp` bit (the bug oven-sh#37074 guards against) makes the redesigned test fail by timeout, and restoring the per-socket gate makes it pass. - Coverage check for the mid-request test's complete variant: dropping `HTTP_CLOSE_WHEN_IDLE` from `HTTP_CONNECTION_SCOPED` (so the mark is wiped when the post-stop request dispatches) makes it fail by timeout; restored, it passes. The updated fixture also runs 30/30 on the macOS 14 CI host under the build 90105 canary, and hangs as expected under a pre-oven-sh#37074 build. The remaining hazard - any long-running wildcard port-0 listener on darwin CI can have its loopback traffic stolen by a later explicit `127.0.0.1` bind such as `VerdaccioRegistry`'s `randomPort()` (range 1024-65535 overlaps the kernel's ephemeral range) - is worth addressing in the harness separately; noted on oven-sh#36352. <!-- robobun:evidence:begin --> --- **no test proof** · iteration 0 · Platform-specific test-only change; deferring to CI. <!-- robobun:evidence:end -->
This was referenced Aug 14, 2026
Closed
Collaborator
Author
|
One more data point for this fix, from a long-lived container rather than CI. Still reproduces on current main (8326d1b): Because every file that does For cross-reference: #34687 carries the same |
Collaborator
Author
|
This leak is now breaking main. In build 108576 In both shards three This branch merges cleanly onto main today (a6c4cc2). I rebuilt the same fix on top of current main in robobun/7d7bb0a5/verdaccio-registry-stop (stop() awaits the child's exit, start() rejects if verdaccio exits before it listens, randomPort() takes a kernel-assigned port). I am not opening a second PR for it. #38891 covers the port side from the other direction. |
Collaborator
Author
|
Three PR builds failed on 2026-10-02 because a fixed port was in use on all four attempts. I reproduced the first one with a registry that
Reproduction of build 122856Linux x64, main at bc7a813. The commands are at the end.
I did not run the two darwin failures. They have the same shape: the listen that failed is on Why it is rare, and why each retry fails
This PR against main today
Order with #43735#43735 moves Commands and run details
import { afterAll, beforeAll, expect, test } from "bun:test";
import { VerdaccioRegistry } from "harness";
const registry = new VerdaccioRegistry();
registry.port = 12347;
beforeAll(async () => {
await registry.start();
});
afterAll(() => {
registry.stop();
});
test("registry answers", async () => {
const res = await fetch(`http://127.0.0.1:${registry.port}/`);
expect(res.status).toBe(200);
// Install test files run for seconds. Give the registry the same time to write its log.
await Bun.sleep(2000);
});On main: $ bun test test/hold.test.ts
(pass) registry answers [2036.79ms]
$ ss -ltnp | grep 12347
LISTEN 0 512 127.0.0.1:12347 0.0.0.0:* users:(("bun",pid=89325,fd=13))
$ ps -eo pid,ppid,args | grep '[v]erdaccio'
89325 1 bun test/node_modules/.bun/verdaccio@6.0.0+d56b95ac411b1cdd/node_modules/verdaccio/bin/verdaccio -c ...
$ BUN_TEST_DRAIN_EVENT_LOOP=1 BUN_FEATURE_FLAG_NO_ORPHANS=1 \
bun run --config=bunfig.node-test.toml test/js/node/test/sequential/test-net-server-bind.js
error: listen EADDRINUSE: address already in use :::12347
at emitError (node:events:50:13)
at emitErrorNextTick (node:net:2196:12)
$ curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:12347/no-deps
200
$ ss -ltnp | grep 12347 # a few seconds later: no output, pid 89325 is gone
$ BUN_TEST_DRAIN_EVENT_LOOP=1 BUN_FEATURE_FLAG_NO_ORPHANS=1 \
bun run --config=bunfig.node-test.toml test/js/node/test/sequential/test-net-server-bind.js
$ echo $?
0With the diff of this PR: $ bun test test/hold.test.ts
(pass) registry answers [2022.32ms]
$ ss -ltnp | grep 12347 # no output
$ BUN_TEST_DRAIN_EVENT_LOOP=1 BUN_FEATURE_FLAG_NO_ORPHANS=1 \
bun run --config=bunfig.node-test.toml test/js/node/test/sequential/test-net-server-bind.js
$ echo $?
0The log pipe. A file that sends the registry 1000 requests leaves no registry behind. The same file with a reader on The 31 files, with and without the diff:
Other candidates for the port in build 122856. I replayed the 22 files that ran before the failed test and listed the TCP listeners after each file. The files that passed left only registries behind, each on In that replay, 5 of the 22 files have failing tests in my container, where the release build is older than the tests. One of them, Timeline of build 122856, shard 16 (UTC): the four registry files ( |
dylan-conway
added a commit
that referenced
this pull request
Oct 13, 2026
### Problem
- The install tests fork verdaccio 6 and assert its answers where npm
differs. `/-/whoami` with an unknown token: verdaccio `200 {}`, npm `401
{}`.
- All 31 files share one `htpasswd` and one storage directory. When
verdaccio cannot listen on its random port, `start()` never settles: 6
of 155 local file runs, and main build 124221.
### Fix
- `test/packages/registry` is an npm registry on `Bun.serve`, without
dependencies. Its state is in memory, per instance.
- The 31 files import `TestRegistry` from `"registry"`.
`VerdaccioRegistry` and the verdaccio dependencies are removed, and
three packages that only verdaccio installed leave the bytecode snapshot
of `libraries.js`.
- `yarn-berry.test.ts` came to main with #40057, after the rebase. It is
migrated too (7025402), so 32 files use `TestRegistry`. Each number
for 31 files is from before that.
- Verified: `test/packages/registry/` (183 tests) and the 31 files (2518
pass, 0 fail).
### Test timings
| 31 files | main, verdaccio | this PR |
|---|---|---|
| local, wall-clock | 158 s | 83 s |
| local, CPU | 214 s | 113 s |
| CI, six lanes | 985 to 1090 s | 549 s |
Same base commit. Details: Notes.
### Background
- #33115 has the same goal. A maintainer has to choose.
- Reads follow registry.npmjs.org: 49 of 63 compared requests agree.
Writes follow the npm docs.
- Considered a class in `harness.ts`: every test file would load it.
### Downsides
- The registry is faster, so packuments arrive in another order. One
test depended on that (54 of 100 local runs failed) and orders them now.
- The registry runs on the test thread: a `spawnSync` of bun against it
never returns.
<details><summary>Notes</summary>
#### What I ran
After the rebase. The head is 9e9e394 on main c6da4a4. Linux x64,
12 vCPU. bun is the canary build e555359, which is 8 commits behind
that main commit.
| what | result |
|---|---|
| the 31 migrated files, 5 rounds, head 2d236bf | 154 of 155 file
runs passed: 2518 pass and 3 skip in a round. One run had one failing
test, which the commit after it repairs |
| the 31 files, 5 more passes and 5 runs in a row of each file, head
9e9e394 | 310 file runs, 0 fail |
| the same 31 files on main with verdaccio, 5 rounds | 2516 pass, 0
fail, 3 skip. 6 of 155 file runs did not start verdaccio |
| `test/packages/registry/` | 183 pass |
| `bundler_bytecode_portable.test.ts` | 22 pass |
| `tsc --noEmit` in `test/` | 7 errors on main, the same 7 on this
branch, none in a file of this PR |
| CI, build https://buildkite.com/bun/bun/builds/124229, head 2d236bf
| 183 of 183 jobs passed |
- The two tests that only this PR has are the difference between 2516
and 2518: the `whoami` test and the test of the publish header.
- The runs are after a clean install of `test/node_modules` from the
lockfile of each tree.
- The canary build passes the bytecode test of main on the tree of main.
So its bytecode is the bytecode of main.
- I did not run Windows or macOS after the rebase. The Windows machine
was not available. CI ran both.
- Build 124229 has one test of this PR that passed on a retry: see the
flaky tests below.
After the merge of main 110efa0 (d947d38) and the migration of
`yarn-berry.test.ts` (7025402). Linux x64. bun is `bun-linux-x64` of
main build 124948, which is 110efa0:
| what | result |
|---|---|
| `yarn-berry.test.ts` on d947d38 | does not load: `Export named
'VerdaccioRegistry' not found` |
| `yarn-berry.test.ts` on 7025402 | 58 pass, 0 fail |
| the same file, 40 runs in a row, then 5 rounds of 8 runs at the same
time | 80 runs, 0 fail |
| the same file with a registry that has no packages | 21 of 58 fail, so
the tests do read the registry |
| `bundler_bytecode_portable.test.ts` | 22 pass |
| `test/packages/registry/` | 183 pass |
- None of the inputs of the bytecode snapshot changed between c6da4a4
and 110efa0, so its values stay.
- The other 31 files were not run again after the merge. CI runs them.
Before the rebase, on the head a4367e5 (then 3b5d5ee):
- CI build https://buildkite.com/bun/bun/builds/121299 passed 181 of 181
jobs.
- Windows x64, canary a4f1429: `test/packages/registry/` had 183 pass.
`bun-publish`, `config-precedence`, `pnpm-lock-v9` and
`bun-update-lockfile-sync` had 258 pass, 0 fail, 1 skip, three commits
earlier.
- A local debug build with ASAN: `test/packages/registry/`, `hoist`,
`npmrc`, `config-precedence`, `bun-publish` and `bun-lockb` had 319
pass, 0 fail, when the package had 165 tests.
- With the leftovers of verdaccio in the fixture directory, the 31 files
had 2499 pass and 1 fail. The failure was `npm manifest cache entries
are only reused for the package name they were saved for`, which is
flaky on main too.
#### The rebase
- The base moved from a4f1429 to c6da4a4, 88 commits. The merge
commit is gone. The 12 commits were replayed, so each has a new id.
- 7 files had a conflict. In 5 of them #44614 changed a line that this
PR deletes or renames. `test/package.json` is the file of main without
the two verdaccio lines.
- `test/bun.lock` is the lockfile of main after `bun install
--lockfile-only` without verdaccio. Its diff has the same 578 lines as
before the rebase.
- After the replay, 42 of the 59 files had the same bytes as before the
rebase. For the other 17, the patch of this PR had the same lines, apart
from lines of main that it deletes and the snapshot.
- The snapshot of `libraries.js` has new values, because main has a new
WebKit: see the bytecode snapshot below.
- Two commits are new. 2d236bf: main changed bun-types and
`test/tsconfig.json` (#44614). With them, four lines of the package had
a type error. `request.bytes()` and `request.clone()` have the types of
undici there, and `BodyInit` is `Bun.BodyInit`.
- 9e9e394: see "A test that depended on the order of two packuments".
#### This PR and #33115
#33115 replaces verdaccio too. I did not find it before I opened this
PR: my search for open pull requests stopped at 20 results. Both PRs
change the same test files, so only one can land as it is. The author of
#33115 compared them in
#33115 (comment).
| | #33115 | this PR |
|---|---|---|
| Files | 1083 | 59 |
| State | conflicts with main, four review passes | applies to main |
| Fixtures | converted to source directories | as they are on main |
| The two mock registries (`dummy.registry.ts`,
`simple-dummy-registry.ts`) | replaced | not touched |
A maintainer has to choose. If this PR lands, #33115 can be closed, or
reduced to the parts that only it has.
**Rulings of the review of #33115 that this PR follows**
- A publish is refused with 415 unless the Content-Type is
`application/json`, character for character. `bun-publish.test.ts`
asserts the header that bun sends. The two comments in
`src/runtime/cli/publish_command.rs` stay as they are.
- An error that a handler throws fails the test: `stop()` throws it.
- The bulk advisory endpoint reads a gzip body, and a test sends one.
- `createTestDir()` does not remove users or tokens.
- `hasInstallScript` is true for a script that is not empty.
- `time.modified` moves on each write.
- A field of an advisory that is passed as undefined keeps its default.
The test compares with literal values.
- Nothing here says that bun reads `Cache-Control`. The registry sends
the header because npm does.
**What only #33115 has**
- The replacement of `dummy.registry.ts` and `simple-dummy-registry.ts`.
18 test files import them.
- `registry.define()`: a package that a test declares in code, with a
tarball writer and its golden files.
- The conversion of the fixtures to source directories.
- `registry-resolver-matrix.test.ts`.
Each of these can go on top of this registry as a PR of its own. So can
the three servers in `bun-audit.test.ts`, `config-precedence.test.ts`
and `bun-install-lifecycle-scripts.test.ts` that forward to the registry
to see or to change a request: `intercept` and `recordRequests` do that
now.
#### How each behaviour was established
**Compared with registry.npmjs.org, GET and HEAD only**
I sent 63 requests to registry.npmjs.org and to this registry with
copies of `is-number` and `@types/is-number`, and compared the status,
nine headers and the body. 49 agree. The requests cover:
- The packument in both forms, `HEAD`, `Range`, `?write=true`, and a
scoped name as `@s%2fn`, `@s/n` and `%40s%2fn`.
- A version by number and by tag, and one that is not there: 404 with
the JSON string `"version not found: 9.9.9"`.
- A tarball: `GET`, `HEAD`, `Range` with 206 and 416, `If-None-Match`,
and the 404 for the scope in the file name.
- `dist-tags`, `collaborators`, `visibility`, `ping`, `whoami`, the
account endpoints, `/-/v1/done`.
- The same endpoints with a token that npm does not know.
Separate probes gave:
- The abbreviated version is an allow list without `libc`,
`description`, `scripts` and `time`. `hasInstallScript` is there only
when true. The census covered 5763 versions of 9 packages.
- `Accept`: the registry looks for the media type anywhere in the
header, in exact case. It does not weigh q.
- The entity tag is the hex md5 of the body. It is weak on an encoded
answer. `If-Modified-Since` has no effect.
- Encoding: brotli when accepted, else gzip. Bodies of 21 and 26 bytes
came back as they are, a body of 63 bytes came back encoded. The
registry here starts at 48 bytes.
- `Last-Modified` is `modified` rounded up to the second. I checked one
packument.
- `You must be logged in to publish packages.` is the text of npm for a
token that it does not know, on `collaborators`.
**The 14 requests that differ**
| request | npm | this registry |
|---|---|---|
| a version, and search (4 requests) | the headers change between a
first and a later request: `public`, `last-modified` and `accept-ranges`
come and go | always `cache-control: max-age=300` |
| a packument or tarball that is not there (3) | `cache-control: public,
max-age=300` for an unscoped name, nothing for a scoped one | no
`cache-control` |
| `collaborators` | names in another order | the order of `maintainers`
|
| `/-/npm/v1/keys` | the keys of npm | `{"keys":[]}`, it does not sign |
| `/-/nonexistent-endpoint`, `/-/v1/login/cli/<id>` | 405 with `allow` |
404 `ResourceNotFound` |
| `/<name>/-rev/<rev>` with GET | `allow: PUT` | `allow: PUT, DELETE` |
| `?write=true` with the abbreviated `Accept` | `content-type:
text/plain` | `application/json` |
| `/login/cli/<id>` | 404, the page is on the website | stands in for
that page |
**From the npm/registry docs and the npm CLI source, not sent to the
live registry**
- Login with `PUT /-/user/org.couchdb.user:<name>`: 201 and `{ok, id,
rev, token}`, or 401 and `{"ok":false}`.
- Tokens and profile under `/-/npm/v1/`.
- The publish document, `www-authenticate: OTP`, the web flow with
`authUrl` and `doneUrl`, 202 with `retry-after`.
- The one-time password message. bun compares against this exact text.
- The request sequences of deprecate, unpublish, owner and dist-tag.
- The bulk advisory request and answer.
**My wording, because I do not know the text of the registry**
- Each `400 Bad Request: ...` for a publish body that fails a check, the
415, and the 403 for a read-only token.
- `409 Document update conflict.` is the text of CouchDB.
- Publish answers `200 {"ok":true,"id","rev"}` and a dist-tag write
answers `{"ok":"dist-tags updated"}`. I recall both from npm output. I
did not verify them.
- A restricted package answers 404 to a client that cannot read it. I
recall this from npm. I could not probe it, because I know no private
package.
- `You cannot publish over the previously published versions`, `You do
not have permission to publish` and `cannot be republished until 24
hours have passed` are the texts of npm as I recall them. I did not
probe them.
- A publish with a token that the registry does not know answers 401
without `www-authenticate`. npm sends `www-authenticate: Basic, Bearer`
on `collaborators`. I do not know what it sends on a publish.
#### Where the registry is more permissive than npm
- Every user can publish into every scope. npm needs a user or an org
with the name of the scope.
- A restricted package needs no paid plan. npm answers 402.
- A package of the storage directory has no maintainers, so every user
can add a version to it.
Not implemented: signatures, provenance, OIDC, orgs and teams.
#### What the tests saw change
Two tests of the 31 files depended on verdaccio. `whoami > invalid
token` asserted its answer, `200 {}`. It asserts `401 Unauthorized` now,
and a new test keeps the `200 {}` case with a mock server. `ranged peer
dependency resolution is stable across installs from bun.lock` depended
on its speed: see the section on the order of two packuments. Every
other test passes unchanged.
`bun-publish.test.ts`:
- Each `rm` of a package directory is a `registry.packages.delete()` of
that name. The next section has the reason.
- `--ignore-scripts` published `publish-pkg-4` and removed
`publish-pkg-5`. It passed only because the test before it removed
`publish-pkg-4`. It removes the right package now.
- `republishing normally fails` matched `/403|409|already exists|already
present|cannot publish/`. It asserts the status and the message.
- A new test asserts that the publish request has `content-type:
application/json`.
`harness.ts` does not export `VerdaccioRegistry`. `REVIEW.md` says to
delete code in the PR that makes it dead. The eight files that had a
variable `verdaccio` keep that name, so their diff stays small.
A bug of the registry that the review found: `bun publish` of a version
with build metadata got a 400. bun sends the key `1.0.0` and a manifest
with `1.0.0+build.5`. The registry accepts that now, and `cli.test.ts`
publishes one with bun.
#### A checkout that ran the tests on verdaccio
Verdaccio wrote each package that a test published into
`test/cli/install/registry/packages`. That is the fixture directory too,
and ignore rules hide what verdaccio wrote. One run of the 31 files on
main left 18 packages and `.verdaccio-db.json` there.
The new registry reads that directory. So it served those packages, and
a publish of the same version got `403 You cannot publish over the
previously published versions`. With the leftovers of that run in place,
15 of the 47 tests of `bun-publish.test.ts` failed.
- `bun-publish.test.ts` calls `registry.packages.delete()` in the 26
places where main removes a package directory. With the same leftovers
in place the file passes.
- The ignore rules stay as they are on main, so the leftovers stay out
of `git status`.
- The builds of this PR did not show this: a checkout of CI has no
leftovers.
#### Layout
The package has the layout of `test/packages/s3-server`, which came to
main while this PR was open: the sources in `src/`, the tests in
`test/`, and the helpers of the tests in `test/helpers.ts`. `index.ts`
does not import the harness. `test-registry.ts` has the class for the
tests of bun, and the alias `"registry"` points to it.
#### One port, one registry
`Bun.serve` sets `SO_REUSEPORT` when it gets `development: false`
(`src/runtime/server/ServerConfig.rs:716`). The registry passed that
option, so a second registry on the same explicit port started with no
error. It passes `reusePort: false` now, and the second one fails with
`EADDRINUSE`. The new test fails without the option.
The tests listen on port 0. On Linux, 20000 listeners on port 0 with
`SO_REUSEPORT` got 20000 distinct ports, so I have no sign that a test
shared a port. I did not check that on macOS or Windows.
#### The bytecode snapshot
`bundler_bytecode_portable.test.ts` pins the bytecode of a bundle of 64
libraries from `test/node_modules`. The bundle holds each package that a
library can load, also one that it loads in a `try` block and does not
declare.
`test/` uses the isolated linker. A package that is not declared is
still found through `node_modules/.bun/node_modules`, which has one
version of each installed package. Two libraries of the bundle load a
package that way:
| library | loads | with verdaccio | without |
|---|---|---|---|
| `mongodb` (`lib/deps.js`) | `aws4` | `aws4@1.13.0`, installed only for
`@cypress/request`, a dependency of verdaccio | not found, so not in the
bundle |
| `follow-redirects` (`debug.js`) | `debug`, an optional peer |
`debug@4.3.4` with `ms@2.1.2`. Only `@verdaccio/config` and
`@verdaccio/local-storage-legacy` resolve to that version |
`debug@4.4.0`, which `superagent` and others already bring |
So 7 modules of 3 packages leave the bundle: `aws4@1.13.0`,
`debug@4.3.4` and `ms@2.1.2`. Nothing is added. To check this, the
lockfile of c6da4a4 and the lockfile of this PR were installed in two
directories, and `libraries.js` was built in each with the command of
the test. The sha256 of the two `.js` outputs are the old and the new
value of the snapshot. A metafile names the two importers above.
The bytecode is 32632 bytes smaller: 21179744, where main has 21212376.
Before the rebase the two numbers were 21174272 and 21206912, with the
WebKit of that time. The new values are the same on every platform,
which is the case where the comment at the top of that file says to
update the snapshot.
#### Test timings
"Before" is main at c6da4a4, where the 31 files fork verdaccio.
"After" is this PR on the same commit.
**Local**
- Linux x64 VM, 12 vCPU (Intel Xeon, 2.90 GHz), 16 GB. bun is the canary
build e555359 for both sides.
- Each file ran alone: `bun test --timeout=90000 <file>`. The two trees
took turns on each file. The order of the two changed with each file and
each round. 5 rounds, 310 file runs.
- Wall-clock is from the start of `bun test` to its exit. CPU is the
user and system time of all processes of the VM in that time, from the
cgroup. It has the verdaccio process, which the test does not wait for.
- The table has the median of the runs that exited with 0, and the
lowest and the highest in brackets. Seconds.
- The after side is the commit 2d236bf. The commit after it changes
one test of `isolated-install.test.ts` and adds no wait for time.
- The VM lost CPU time to other VMs in each round (steal time), so the
wall-clock has noise. The rounds agree: before is 1.7 to 2.1 times after
in each one.
| file | tests | before, wall | after, wall | ratio | before, CPU |
after, CPU |
|---|---|---|---|---|---|---|
| `bun-add-catalog.test.ts` | 149 | 4.81 (4.34 to 6.17) | 2.15 (1.88 to
3.47) | 2.2 | 6.94 | 3.16 |
| `bun-add-filter.test.ts` | 123 | 4.67 (3.81 to 7.01) | 1.91 (1.84 to
2.19) | 2.4 | 6.75 | 3.10 |
| `bun-audit.test.ts` | 182 | 6.79 (6.43 to 9.03) | 5.21 (4.62 to 5.39)
| 1.3 | 10.43 | 6.83 |
| `bun-dedupe.test.ts` | 82 | 4.31 (4.10 to 4.74) | 2.19 (1.99 to 2.37)
| 2.0 | 6.96 | 3.61 |
| `bun-install-lifecycle-scripts.test.ts` | 128 | 10.40 (8.70 to 14.14)
| 8.51 (7.50 to 9.92) | 1.2 | 21.07 | 16.04 |
| `bun-install-native-binlink.test.ts` | 16 | 3.93 (3.71 to 6.41) | 1.79
(1.40 to 5.45) | 2.2 | 5.40 | 3.04 |
| `bun-install-patch.test.ts` | 31 | 3.93 (3.77 to 4.66) | 2.52 (2.04 to
2.88) | 1.6 | 4.15 | 2.09 |
| `bun-install-registry.test.ts` | 282 | 18.79 (16.54 to 26.41) | 15.38
(10.74 to 16.10) | 1.2 | 18.89 | 12.45 |
| `bun-lock.test.ts` | 40 | 3.98 (3.40 to 4.51) | 1.33 (1.27 to 1.59) |
3.0 | 4.48 | 1.61 |
| `bun-lockb.test.ts` | 8 | 2.29 (1.90 to 2.39) | 0.53 (0.43 to 0.64) |
4.3 | 2.45 | 0.52 |
| `bun-patch.test.ts` | 37 | 4.16 (4.05 to 6.92) | 3.10 (2.44 to 3.78) |
1.3 | 5.02 | 3.18 |
| `bun-pm-licenses.test.ts` | 80 | 5.16 (3.53 to 5.78) | 2.19 (1.53 to
2.36) | 2.4 | 6.61 | 2.91 |
| `bun-prune.test.ts` | 121 | 5.40 (5.14 to 5.63) | 2.80 (2.62 to 3.46)
| 1.9 | 8.18 | 4.38 |
| `bun-publish.test.ts` | 47 | 5.40 (5.03 to 6.02) | 2.30 (2.00 to 3.30)
| 2.3 | 5.42 | 2.15 |
| `bun-update-lockfile-sync.test.ts` | 77 | 4.22 (3.39 to 4.68) | 1.78
(1.52 to 2.15) | 2.4 | 5.95 | 2.78 |
| `bun-update-transitive.test.ts` | 224 | 8.46 (7.73 to 10.16) | 5.06
(4.44 to 8.29) | 1.7 | 13.50 | 8.56 |
| `bun-update.test.ts` | 159 | 8.28 (7.66 to 8.69) | 4.59 (4.27 to 5.02)
| 1.8 | 11.96 | 6.73 |
| `bun-workspaces.test.ts` | 82 | 3.56 (3.22 to 3.75) | 1.59 (1.48 to
1.77) | 2.2 | 4.85 | 2.42 |
| `catalogs.test.ts` | 89 | 5.90 (4.93 to 6.02) | 2.04 (2.00 to 2.20) |
2.9 | 7.00 | 3.28 |
| `config-precedence.test.ts` | 51 | 3.65 (2.81 to 4.11) | 0.90 (0.74 to
0.97) | 4.1 | 4.46 | 1.12 |
| `frozen-lockfile-missing-workspace.test.ts` | 1 | 1.79 (1.64 to 2.12)
| 0.15 (0.14 to 0.18) | 11.9 | 2.02 | 0.18 |
| `frozen-lockfile-pruned.test.ts` | 101 | 4.76 (4.05 to 5.96) | 1.83
(1.71 to 1.95) | 2.6 | 7.05 | 3.23 |
| `hoist.test.ts` | 1 | 1.75 (1.71 to 1.91) | 0.17 (0.12 to 0.21) | 10.3
| 2.09 | 0.17 |
| `isolated-install.test.ts` | 85 | 6.24 (5.94 to 7.74) | 3.59 (3.15 to
3.95) | 1.7 | 7.67 | 4.32 |
| `isolated-relink.test.ts` | 6 | 2.29 (1.89 to 2.96) | 0.35 (0.30 to
0.47) | 6.5 | 2.72 | 0.48 |
| `migration/pnpm-lock-v9.test.ts` | 84 | 3.86 (3.64 to 4.85) | 1.54
(1.34 to 1.83) | 2.5 | 4.57 | 1.89 |
| `migration/pnpm-migration.test.ts` | 3 | 1.89 (1.72 to 2.16) | 0.18
(0.16 to 0.22) | 10.5 | 2.29 | 0.23 |
| `nested-overrides.test.ts` | 144 | 8.95 (8.20 to 10.21) | 5.92 (4.50
to 8.86) | 1.5 | 15.03 | 9.69 |
| `npmrc.test.ts` | 47 | 2.52 (2.32 to 2.68) | 0.58 (0.55 to 0.61) | 4.4
| 2.91 | 0.65 |
| `public-hoist-pattern.test.ts` | 14 | 2.53 (2.44 to 2.62) | 0.46 (0.44
to 0.48) | 5.5 | 3.01 | 0.60 |
| `update_interactive_formatting.test.ts` | 24 | 2.99 (2.79 to 3.28) |
0.70 (0.55 to 0.99) | 4.3 | 3.93 | 1.11 |
| **sum** | **2518** | **157.7** | **83.3** | **1.9** | **213.8** |
**112.5** |
Each round, the sum over the files where both sides exited with 0:
| round | files | before, wall | after, wall | before, CPU | after, CPU
|
|---|---|---|---|---|---|
| 1 | 29 | 146.4 | 71.2 | 184.7 | 95.8 |
| 2 | 30 | 140.9 | 68.0 | 188.4 | 98.6 |
| 3 | 31 | 165.4 | 84.1 | 217.1 | 115.2 |
| 4 | 29 | 158.3 | 80.0 | 202.1 | 106.5 |
| 5 | 29 | 150.3 | 88.1 | 209.3 | 114.6 |
Runs that are not in the tables:
- 6 of the 155 runs with verdaccio did not start it. The log has
`Verdaccio exited with code 2 and signal null`. Code 2 is the exit of
verdaccio when it cannot listen (`build/lib/bootstrap.js`). `start()`
waits for a message that does not come. Five runs ended at the hook
timeout of 90 s. `bun-install-registry.test.ts` waits at module scope,
where no timeout applies: I stopped it after 678 s.
- I started verdaccio on a port that was in use, to check: the same
line, and `start()` did not settle.
- 1 of the 155 runs of this PR had one failing test: see the next
section.
**CI**
Build 124221 is main at c6da4a4. Build 124229 is this PR on the same
commit. Builds 124209 and 124201 are main, one and four commits earlier,
with the same 31 files. Each number is the sum over the 31 files of the
duration that the runner logs for a file. I read the logs with the
parser of `scripts/update-test-durations.mjs`. Seconds.
| lane | main 124221 | main 124209 | main 124201 | this PR 124229 |
|---|---|---|---|---|
| debian 13 x64 | 222.6 | 131.4 | 126.7 | 51.5 |
| ubuntu 25.04 x64 | 126.1 | 126.0 | 126.4 | 39.1 |
| alpine 3.23 x64 | 203.8 | 172.8 | 156.9 | 119.8 |
| debian 13 aarch64 | 141.7 | 145.6 | 141.2 | 45.6 |
| Windows 2019 x64 | 149.6 | 166.2 | 171.6 | 109.9 |
| Windows 11 aarch64 | 245.8 | 243.4 | 264.1 | 182.8 |
| **sum of the six lanes above** | **1089.6** | **985.4** | **986.9** |
**548.7** |
| debian 13 x64 ASAN | no tests | no tests | no tests | 340.2 |
| macOS aarch64 | no tests | no tests | no tests | 53.1 |
| macOS x64 | no tests | no tests | no tests | 67.1 |
- The 222.6 s of build 124221 have one run of 90 s:
`isolated-install.test.ts` on shard 8. The log has `Verdaccio exited
with code 2 and signal null` and the hook timeout. The second attempt
passed.
- The main builds have no test jobs for ASAN and macOS.
- A build is one sample. Apart from the 90 s run, the three main builds
differ by up to 47 s on a lane (alpine).
- Not each file is faster in CI. On each lane, 1 to 6 of the 31 files
took longer than in each of the three main builds. The largest
differences are on alpine: `bun-install-registry.test.ts` took 37.3 s
against 27.9 to 29.2 s, and `bun-update.test.ts` took 7.2 s against 2.5
to 2.7 s. That is one sample, and I do not know the cause. Locally each
file is faster.
- The four test files of `test/packages/registry/` take 1.5 to 4.8 s on
a lane.
Import cost, CPU, the lowest of the runs. Measured before the rebase, on
a build of 367d939:
| | release | debug |
|---|---|---|
| `harness.ts` | 29.5 ms | 1378 ms |
| the registry | 21.5 ms | 1162 ms |
| `node:zlib` alone | 11.5 ms | 615 ms |
#### A test that depended on the order of two packuments
`isolated-install.test.ts`, `ranged peer dependency resolution is stable
across installs from bun.lock`. The root depends on
`normal-dep-and-dev-dep` (which has `no-deps@1.0.1`), on
`two-range-deps` (`no-deps@^1.0.0`) and on `peer-deps-fixed` (peer
`no-deps@^1.0.0`). The test needs 1.0.1 and 1.1.0 in the graph.
- bun resolves `^1.0.0` to 1.1.0 when the packument of `two-range-deps`
is there first. When the packument of `normal-dep-and-dev-dep` is there
first, 1.0.1 is in the graph and `^1.0.0` resolves to it.
- I delayed one packument by 40 ms, with 12 installs for each case. A
slow `two-range-deps` gave one version 12 times. A slow
`normal-dep-and-dev-dep` gave both versions 12 times. No delay gave both
7 times and one 5 times.
- On main with verdaccio the test passed 50 of 50 runs. With the new
registry it failed 54 of 100 runs alone, and 10 of 20 runs of the file.
It fails with the older bun build 367d939 too (15 of 40), so the
rebase did not bring it.
- CI did not show it, before and after the rebase. CI sets
`BUN_INSTALL_CACHE_DIR`, so the tests of a file share one manifest
cache. With that variable the file passed 10 of 10 local runs. Without
it each test has its own cache.
- My local runs before the rebase did not show it. They were on a host
with a load average of 400 to 900.
- The test has its own registry now. Its `intercept` answers for
`normal-dep-and-dev-dep` after bun asked for `no-deps`, which bun does
when it has the packument of `two-range-deps`. There is no wait for
time. 100 of 100 runs alone and 20 of 20 runs of the file pass.
- After that change: 5 passes over the 31 files, and 5 runs in a row of
each file. 310 file runs, 0 failures.
Other tests can depend on the order in the same way. `hoisting > peers >
it should hoist 1.0.1 when peer *` does: see the flaky tests below. The
cause is in `bun install`, not in the registry: one `package.json` gives
two different trees.
#### Other things I found
- Before the rebase, `test/bun.lock` on main was stale: `bun install
--lockfile-only` on an unchanged main dropped five nested `react`
entries, and I kept them by hand. The lockfile of main is current now.
The lockfile diff here is what bun writes: verdaccio and its
dependencies go, and `debug` hoists to 4.4.0.
- `manifest conditional requests > a changed etag returns 200 and the
new etag is cached` failed once on unchanged main and passed on the next
run. It uses its own mock server, not the registry. It is in
`test/flaky-tests.txt`.
- On my local debug build, `Bun.serve` adds `content-type:
application/octet-stream` to an answer that has no body: a 304, and a
401. Two release builds, one of 367d939 and one of 36cd151, do not.
I did not find the cause and I saw it on one local build only. The tests
here assert the exact headers on the `Response` of the registry, and on
the wire only the headers that the registry sets.
- `Bun.serve` sets `SO_REUSEPORT` when it gets `development: false`. The
type of `reusePort` says that the default is false.
- `bun install` gives another tree for the same `package.json` when the
packuments arrive in another order: see the section on the order of two
packuments.
- `bun run` and `bun test` use a whole core, and do not end, while a
top-level await waits for a promise that nothing settles. A file with
`await new Promise(() => {})` used 9.9 s of CPU in 10 s. node prints
`Warning: Detected unsettled top-level await` and exits with 13. The
loop is `wait_for_promise` in `src/jsc/event_loop.rs`. This is how
`bun-install-registry.test.ts` on main hangs when verdaccio does not
start.
#### Flaky tests in files that this PR edits
Two tests of migrated files passed on a retry in the earlier builds of
this PR. Main has both without this change. In build 121299, the last
one before the rebase, no file of this PR was in the flaky annotations.
Files outside this PR were.
- `bun-install-registry.test.ts`, `hoisting > peers > it should hoist
1.0.1 when peer *`, on Windows 11 aarch64, with `a-dep@1.0.9`. The same
test fails the same way on the same lane in the main builds
https://buildkite.com/bun/bun/builds/120426 and
https://buildkite.com/bun/bun/builds/120659. I could read 11 main
builds. After the rebase it passed on a retry on ubuntu 25.04 aarch64 in
build 124229, again with `a-dep@1.0.9`. The version that it gets depends
on the order of the packuments.
- `bun-lock.test.ts`, `peer no published version satisfies > declared by
a registry package`. It runs against its own mock server.
`test/flaky-tests.txt` lists it at 34 of 80 builds.
I cannot say that the rate of the first one is unchanged. On my machine
it failed 1 time in 150 runs with this registry, and 0 times in 29 runs
with verdaccio.
A third test failed in my runs, and not in the builds of this PR:
- `bun-install-registry.test.ts`, `npm manifest cache entries are only
reused for the package name they were saved for`: `failed to open
manifest file ... ENOENT`. It failed in 1 of 8 runs of the file. Alone,
it failed in 2 of 40 runs with this registry, and in 5 of 40 runs on
main with verdaccio. `bun install` removes a manifest file that is not
valid and saves the new one in a task of the thread pool
(`Serializer::save_async` in `src/install/npm.rs`). I found nothing that
waits for that task before the exit. The test reads the file after the
exit. `test/flaky-tests.txt` has `failed to open ...` for this file.
#### Reviews of the diff
My review of the diff raised 9 concerns. 8 were addressed in full: the
header of a publish, an error in a handler, the export of the old name,
`pack()`, the claims about npm, #33115, the flaky tests, and the time of
the tests in `cli.test.ts`. One is addressed in part: the review named
six follow-ups, and these Notes list the three that I can describe.
The reviews on GitHub raised 12 more. All are addressed:
- The URL of a registry names the host that it listens on. It was
`localhost` for each host.
- A checkout with the leftovers of verdaccio failed the publish tests:
see the section above.
- The ignore rules for the leftovers stay.
- `intercept` gets a copy of the request, so it can read the body of a
request that the registry answers.
- The web login test fails with the output of bun when bun exits before
it asks for the session. Before, it waited for the timeout.
- Each test of the package disposes its registry when an assertion
fails.
- `VerdaccioRegistry` is removed. My review asked to keep the export for
open PRs. The review on GitHub asked to remove it, and `REVIEW.md` says
the same.
- A request that the registry refused left a part of its change in the
package: a `PUT` to `/<name>/-rev/<rev>` with a tag that is not valid,
and a deprecation with a message that is not a string. The checks come
before the first change now.
- In a search, a qualifier that matched took away the miss of a word
before it. A package is found when each word matches.
- A storage that is not a directory gave a registry that answered 404
for every package. The constructor throws now, and `cli.ts` prints the
message. A read that failed does not stay as the state of the package.
- `cli.ts` ended with a stack trace for a `--user` that the registry
refuses, for an option that it does not know, and for a port that is in
use. One handler prints the message and the usage for each refusal now.
- `cli.ts` took an empty `--port` as port 0, and it took `0x50` and
`1e3`. The value must be digits now.
One more finding got no change: a scope directory that another process
replaces with a file while the registry reads it. Nothing writes into
the storage while a registry runs, and a check before the read has the
same window. The reviewer withdrew the finding.
#### Related open PRs
These change `VerdaccioRegistry` in `test/harness.ts`. This PR removes
that class, so git reports a conflict, and the problem each one
addresses is not there with the new registry:
- #36352: `stop()` does not end the process. The new `stop()` closes the
server.
- #38891: the random port and a verdaccio that exits early. The new
registry listens on port 0 in the process of the test.
- #40219: `createTestDir()` removes the users. The new registry keeps
them.
- #40766: run verdaccio under node in CI.
These edit the line of `hoist.test.ts` that imports `VerdaccioRegistry`.
Git reports a conflict on that line. The new line imports `TestRegistry`
from `"registry"`:
- #43878
- #38976
I looked at the 500 open PRs that changed last. 72 of them change a file
under `test/cli` or `test/regression`, the harness, or a test with
`install`, `publish` or `registry` in its path. I read the diff of each.
No PR other than the ones above adds or changes a line with
`VerdaccioRegistry`.
</details>
<!-- robobun:evidence:begin -->
---
**no test proof** · iteration 0 · platform-specific test(s) that do not
run on this machine, deferring to CI, which covers all platforms:
test/cli/update_interactive_formatting.test.ts,
test/cli/install/migration/migrate.test.ts,
test/cli/install/isolated-relink.test.ts,
test/cli/install/isolated-install.test.ts,
test/cli/install/frozen-lockfile-pruned.test.ts,
test/cli/install/bun-workspaces.test.ts,
test/cli/install/bun-publish.test.ts,
test/cli/install/bun-prune.test.ts, test/cli/install/bun-patch.test.ts,
test/cli/install/bun-lockb.test.ts, test/cli/install/bun-lock.test.ts,
test/cli/install/bun-install-registry.test.ts,
test/cli/install/bun-install-patch.test.ts,
test/cli/install/bun-install-native-binlink.test.ts,
test/cli/install/bun-install-lifecycle-scripts.test.ts,
test/cli/install/bun-add-filter.test.ts,
test/bundler/bundler_bytecode_portable.test.ts
<!-- robobun:evidence:end -->
---------
Co-authored-by: Dylan Conway <dylan.conway567@gmail.com>
Collaborator
|
This is Claude (an AI assistant) writing on Jarred's behalf. Obsolete: verdaccio is gone from the tests (#44055). |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
VerdaccioRegistry.stop()intest/harness.tscallsthis.process?.kill(0). In Node'schild_processAPI, passing signal0tokill()only probes whether the process exists; it does not send a signal. As a result, the forked verdaccio process survives everyafterAll()teardown and leaks for the rest of the test run.This was introduced in #16540, which changed
kill()tokill(0).Reproduction
Observed in practice as stacks of stale
verdaccioprocesses after running install test files locally.Fix
Call
kill()with no argument so the defaultSIGTERMis sent, restoring the pre-#16540 behavior. Now that the process actually exits on teardown, theexitlistener would logVerdaccio exited with code null and signal SIGTERMfor every test file; a#stoppedflag suppresses that message when the exit was initiated bystop(), keeping normal teardown quiet while still surfacing unexpected mid-test crashes.Verification
New test
test/cli/install/verdaccio-registry-lifecycle.test.tsstarts a registry, callsstop(), and asserts the child'sexitevent fires.Before the fix:
After: passes in ~1s,
psshows no lingering verdaccio.Note: the change is entirely under
test/, so the automated fail-before check (which stashessrc/) cannot distinguish before/after. The fail-before output above was captured by revertingkill()back tokill(0)with the new test in place.[stamp-90s] gate passed · iteration 0 · 2 files touched
passes on PR (with fix)
diff hotspot
gate history · 1 passed · 0 rejected · iteration 0
evidence per changed file