Repository navigation
test(install): compare stderr exactly and pin package.json and bun.lock in bun-add-catalog.test.ts - #40327
test(install): compare stderr exactly and pin package.json and bun.lock in bun-add-catalog.test.ts#40327robobun wants to merge 2 commits into
Conversation
…ck in bun-add-catalog.test.ts Every success case now compares stderr to the exact text bun prints once the two progress lines are stripped: "Saved lockfile", nothing, or the note lines followed by "Saved lockfile". Error cases compare the whole message. expectFiles checks every package.json and the workspaces, catalog, catalogs and packages sections of bun.lock against them. The nine --frozen-lockfile re-checks that followed a plain install with nothing to save are gone: that install already proved the lockfile is in step with package.json. 272 spawns become 263.
|
Important Review skippedReview was skipped as selected files did not have any reviewable changes. 💤 Files selected but had no reviewable changes (1)
⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: ASSERTIVE Plan: Pro Run ID: 📒 Files selected for processing (1)
You can disable this status message by setting the Use the checkbox below for a quick retry:
Comment |
|
Updated 4:28 AM PT - Aug 24th, 2026
✅ @robobun, your commit 9510c292bb9b9e2e609ba898e1c674c9264a7293 passed in 🧪 To try this PR locally: bunx bun-pr 40327That installs a local version of the PR into your bun-40327 --bun |
|
Status: the rewrite passes locally, How the time was measured: the x64-asan log of build 104578 (shard position 7/289) shows the |
Windows prints the workspace path in the listing with a backslash.
There was a problem hiding this comment.
I reviewed this PR and didn't find any bugs. Because it's a large test refactor (~800 lines across 149 cases) that both strengthens assertions and removes 9 --frozen-lockfile spawns as redundant, and the Windows CI result mentioned in the description is still pending, a human look would still be worthwhile.
What was reviewed:
expectFiles/lockView/catalogsOf/lockRowderive the expected bun.lock shape from the package.json inputs; checked that the workspace-vs-top-level catalog precedence and empty-object filtering match how the tests use them.- The removed
frozen()calls each follow aninstallSavesNothing/installKeepsLockfilethat already proves bun.lock is accepted unchanged;frozenOk()is kept where no such install precedes it. PROGRESS_LINESstripping andsummary()header/timing regexes — anchored per-line so partial matches can't swallow real output;--dry-runstdout normalizes backslashes for Windows.
Extended reasoning...
Overview
This PR rewrites the assertion helpers in test/cli/install/bun-add-catalog.test.ts (test-only, no runtime code). It replaces expectOk (loose not.toContain("error:")) with expectSaved/expectUnchanged (exact stderr match after stripping the two progress lines), and replaces per-field lockfile checks with expectFiles, which compares every package.json in full and derives the expected workspaces/catalog/catalogs/packages sections of bun.lock from those files plus a caller-supplied resolution map. It also pins several stdout summaries (dry-run, lockfile-only, no-save, remove, update --latest) and exact error messages. Nine --frozen-lockfile spawns that immediately followed a no-op install are removed as redundant; the remaining ones are wrapped in frozenOk().
Security risks
None. Test-only change to a CLI install test file; no auth, crypto, or untrusted-input parsing is touched.
Level of scrutiny
Medium. It's test-only, so it can't break users, and the direction is toward stricter assertions (exact stderr, full lockfile structure) — which is exactly what REVIEW.md asks for. But the diff is large (~800 lines across 149 cases), it deliberately removes some existing checks with a stated justification, and it introduces non-trivial helper logic (catalogsOf picks between workspaces.catalog and top-level catalog; lockRow projects the six keys bun.lock keeps per workspace). REVIEW.md explicitly flags "never silently weaken, skip, or delete an existing test or safety net" — the deletions here are stated and justified, but a maintainer should confirm they agree the installSavesNothing check genuinely subsumes the removed --frozen-lockfile runs.
Other factors
- The author's own status comment says a Windows x64 run is still in progress. Several new exact-string assertions (the dry-run listing, the frozen-lockfile error notes, the code-frame regex) are the kind that can differ by platform; the second commit already handles the dry-run path separator, but confirming the Windows shard is green before merge seems prudent.
- The removed
runBunInstallharness helper (6 built-in expects per call) is replaced by directspawnBun+ explicit checks. That's fine — the new checks are stricter — but it's a pattern change worth a maintainer's nod. - No prior human or bot review comments on this PR beyond CI status. The bug-hunting system found nothing.
|
The Windows x64 run is in now: 149 pass in 9.3s on a debug build of this branch (Server 2019). The one platform difference it found, the backslash in the |
|
Closing after a self-review of the premise. #39549 did the same work on this file six days ago, measured no gain in CI, and closed with the conclusion that the assertion rewrite alone is not worth a diff of this size without a maintainer asking for it. Nothing has changed since, so the same conclusion applies here. The numbers:
What is left is the assertion rewrite. The reusable part of it, stripping the two progress lines and comparing stderr exactly, belongs in For the record, three things the exact assertions showed:
The branch stays available for reference. |
Problem
test/cli/install/bun-add-catalog.test.tstakes 14s on debian 13 x64-asan (build 104578): half is the verdaccio start as the first registry user in its shard, half is 272bunchildren run 5 at a time (the ASAN--max-concurrencydefault,src/options_types/context.rs:506). It is alreadydescribe.concurrentthroughout.not.toContain("error:")and one or two fields of package.json and bun.lock. A new note, a stale row or a moved resolution passed.Fix
spawnBunstrips the two progress lines. Success cases then compare stderr exactly:Saved lockfile, nothing, or the note lines before it. Error cases compare the whole message.expectFiles(89 sites) compares every package.json and derives the bun.lockworkspacesrows and catalog objects from them. Each case adds the resolution of every dependency, so a nested second version (pkg2/no-deps) is visible.installSavesNothingalso pins the(no changes)summary. Tarball cases useinstallKeepsLockfile: every install re-extracts a tarball dependency (bun install always reinstalls vendored packages #8260, fix in install: skip reinstalling file:/URL tarball deps on second install #35576). The 9--frozen-lockfileruns right after such an install are gone: it already showed bun.lock is in step. 272 spawns become 263.bun bd test test/cli/install/bun-add-catalog.test.ts, 149 pass in 5 runs. Wall time is unchanged within noise (Notes): no speedup is claimed.Background
workspaces, the root's catalog objects, and onepackagesrow per resolved package, keyed by its hoisted path (pkg2/no-depsfor a second version nested under a workspace).bun installprintsSaved lockfileonly when the bytes of bun.lock changed.Notes
CI timeline (build 104578, shard 01a03209-fdf7-4f04-a162-576b9c284b16, position 7/289). Header at +0.07s, first dots at +7.12s, summary at +13.95s. The
coordinator: minio/mysql_tls/postgres_tls/mysql_plain readylines land inside the first window. Across the 20 x64-asan shards the first verdaccio file has 5.2s to 10.5s before its first dot, later ones 2.4s to 3.0s. Spawn arithmetic: 272 children / 5 in flight = 54 rounds, 6.8s / 54 = 125ms per child, which is about what a release ASANbun addcosts.Local measurements (debug ASAN, 16 cores, verdaccio under the release bun). Old and new alternated on the same machine: 14.39s/15.12s against 14.33s/14.99s.
registry.start()1.1s. A singlebun addin isolation is 190ms to 210ms cold, 150ms with a warm cache,bun --version105ms, so process start is most of a spawn.--max-concurrency=200finishes in 9.5s but times out most cases at 5s: the work is CPU bound. Sum of per-test durations 53.9s with 6 children in flight at most. Each spawn removed saves about 25ms of CI wall time, so the 9 removed frozen runs are about 0.2s. Removing the pre-installs or the post-install checks would cut more, but each of them is a scenario (an existing lockfile, or a lockfile a later install accepts), so none was removed.What the stricter assertions surfaced. (1) The
note: catalog entry ... changedline, printed whenever an explicit version replaces an entry; with(also used by <member>)when other members reference it. (2) A plainbun installafter any tarball--catalogadd prints1 package installedevery time, in a plain project too (probed with a non-workspace package.json):verify_package_json_name_and_versioncompares the installedversionwith the formatted resolution, which is the URL (src/install/PackageInstaller.rs,resolution.fmt). That is #8260; #35576 fixes it, after which the 11installKeepsLockfilesites can becomeinstallSavesNothing. (3)bun update --lateston acatalogs.defaultentry prints only1 package installed; #38763 adds the update row and will update that line. (4) Hoisting after an add: the added tarball or range is hoisted and the version another member already had moves under that member (pkg2/no-deps), also for--only-missing.Assertion changes.
expectOk(not.toContain("error:"),not.toContain("panic:")) is replaced byexpectSaved(108 sites) andexpectUnchanged(8 sites: re-runs and adds that find the entry and reference already in place). Notes are pinned with their order: pkg1 before pkg2 in the two-note case. The--dry-runlisting, the--lockfile-onlysummary (Saved bun.lock (3 packages)),--no-save(installed no-deps@2.0.0/2 packages installed),bun remove(donewhile pkg2 still uses it,1 package removedafter),bun update --latestand--silent(empty stdout and stderr) pin stdout. The frozen failure pins the error and both notes. The both-catalogs error pins the code frame shape and theat .../package.json:line:colline. The expect() count drops from 1856 to 1274 because onetoStrictEqualon the lockfile view replaces several per-field checks and the harnessrunBunInstall(6 expects per call) is no longer used.Left minimal on purpose. "alias@npm:pkg is written verbatim, like plain add" (#39774 rewrites its expectations) and "name only in peerDependencies, adding with --dev, matches plain add" (#36284 does the same) keep their original shape; only
expectOkbecameexpectSaved.Also run.
bunx prettier --checkclean.tsc -p test/tsconfig.jsonreports nothing for this file. Windows x64 (Server 2019, debug build of this branch): 149 pass in 9.3s. The only platform difference found was the--dry-runlisting, which prints the workspace path with a backslash; the test compares it with forward slashes.