Skip to content

bun test: count a failure that is not a finished test only through fail_file - #44436

Closed
robobun wants to merge 2 commits into
robobun/3eab222d/junit-non-test-failuresfrom
robobun/3eab222d/test-fail-file
Closed

robobun wants to merge 2 commits into
robobun/3eab222d/junit-non-test-failuresfrom
robobun/3eab222d/test-fail-file

Conversation

@robobun

@robobun robobun commented Oct 2, 2026

Copy link
Copy Markdown
Collaborator

Behaviour change: none

Problem

Fix

  • The two counters become private fields of Summary, in a module in jest.rs. Three writers remain: count_failed_test, merge_worker_counts and fail_file(kind, site, detail).
  • fail_file counts what the kind counts, then records it. The six sites call it. A file no worker took is the one kind with no record.
  • Verified: stdout, stderr, XML and exit code equal bun test --reporter=junit: record a failure that is not a finished test #44435 on 8 fixtures. The first commit adds tests that pin the counts and pass without the refactor.
  • Self-reviewed: 3 concerns raised, 3 addressed.

Background

  • Summary holds the totals bun test prints. fail and the count of errors outside a test decide exit code 1.
  • With --parallel, a coordinator hands files to workers. It counts a file itself only when the worker cannot send its totals.
  • This PR is optional and a draft. REVIEW.md asks for agreement before a new cross-module abstraction. Close it if bun test --reporter=junit: record a failure that is not a finished test #44435 is enough.
  • Considered: public counters. Then the next site can repeat the bug.

Downsides

  • Summary grows from 28 to 32 bytes.
  • None found on the per-test path: 1,607 instructions per finished test in the test runner before and after. Release text is 256 bytes smaller (size).
Notes
  • This is the last PR of the stack: bun test --reporter=junit: fill <failure> for a test that fails with a build or resolve error #44434, bun test --reporter=junit: record a failure that is not a finished test #44435, then this one. It changes no output. bun test --reporter=junit: record a failure that is not a finished test #44435 works without it.
  • What each kind of FileFailure counts: Load adds 1 to fail, and 1 to the errors outside a test when the rejection was printed as an unhandled error. DescribeCallback and Unhandled add 1 to the errors outside a test. WorkerCrashed, Aborted and NotDispatched add 1 to fail. NotDispatched has no testcase, so a runner that reads the report runs the file again (ci: run the fast, non-flaky test files as one bun test --parallel batch per shard #36175).
  • The comparison with bun test --reporter=junit: record a failure that is not a finished test #44435 used release builds of both tips. Fixtures: a 5-file repro (one good file, a top-level throw, a syntax error, a missing import, a throwing describe) run plain, with --rerun-each=2 and with --bail=1. A worker that exits under --parallel=2. A worker panic under --parallel=2. Two good files under --parallel=2. Two files that both fail to load. A hook that leaks a rejection. Durations, the hostname and the version were masked. Result: 8 stdout lines, 230 stderr lines, 8,812 XML bytes and 8 exit codes, all equal.
  • Instructions per finished test in the test runner, before and after: 1,607 with no reporter, 2,412 with --reporter=junit, 1,922 in a --parallel worker. The count is the number of guest instructions qemu-user executes inside the functions of the test runner, for 1,100 empty tests minus 100, divided by 1,000. perf and valgrind are not installed on the machine I used.
  • mimalloc entry calls with the reporter on are equal before and after: 50,901 for 1,100 passing tests, 10,029 for 200 throwing describe blocks, 10,827 for 200 unhandled errors.
  • Release text (size): 80,713,299 before, 80,713,043 after.
  • No write to the two counters exists outside the module (grep). A scratch self.reporter.summary().fail += 1 in Coordinator.rs fails with E0616.
  • The first commit pins: the --rerun-each counts, exit code 1 for an error outside a test, and the counts when no worker starts.
  • The three concerns of the self-review: the chokepoint needs a maintainer's agreement, it did nothing useful before the records existed, and its renames touch the module that bun:test: remove unsafe from the test runner #40235 rewrites. So it is last in the stack, a draft, and the only PR with the renames.
  • Suites run on the debug build: junit.test.js, parallel-non-test-failures.test.ts, runner-junit.test.ts, parallel-startup-failure.test.ts, 26851.test.ts, bun_test.test.ts, test-test.test.ts. parallel.test.ts with --timeout 120000: 43 pass, 2 fail. Both are timing tests (work-stealing balances an uneven directory split, each worker has a unique JEST_WORKER_ID and BUN_TEST_WORKER_ID). The machine had a load average above 300. On a release build: 44 pass, 1 fail (--parallel --dots prints one status character per test, which fails in 4 of 8 runs on main and in 2 of 8 here).

No test asserted these counts:
- --rerun-each: an error outside a test counts on every run, a file that
  fails to load counts once.
- exit code 1 when the only failure is an error outside a test.
- --parallel: files left when no worker starts count as failed.
…il_file

The two counters that fail a run, Summary.fail and the count of errors
outside a test, become private fields of Summary inside a small module
in jest.rs. The count of errors moves from TestRunner into Summary.

Three writers remain:
- count_failed_test, for a finished test.
- merge_worker_counts, for the totals a --parallel worker sends for a
  file. The records of those failures arrive as TestDone frames.
- fail_file(kind, site, detail), for a load failure, an error outside a
  test, a crashed worker, a file aborted by a worker panic and a file no
  worker took. It counts what the kind counts and then hands the record
  to the reporter.

The six sites that counted such a failure and recorded it themselves now
call fail_file. A file no worker took is the one kind with no record.
Console output and the JUnit document do not change.
@robobun

robobun commented Oct 2, 2026 •

Copy link
Copy Markdown
Collaborator Author
Updated 3:02 AM PT - Oct 2nd, 2026

❌ @robobun, your commit 251204b has 1 failures in Build #122789 (All Failures):


🧪   To try this PR locally:

bunx bun-pr 44436

That installs a local version of the PR into your bun-44436 executable, so you can run:

bun-44436 --bun

@robobun

robobun commented Oct 2, 2026

Copy link
Copy Markdown
Collaborator Author

Status: draft. Last PR of a stack: #44434, #44435, #44436 (this one). It is optional and changes no output. It needs a maintainer's yes to the chokepoint (fail_file and the private counters) before it leaves draft.

How I checked it: stdout, stderr, the JUnit XML and the exit code equal #44435 on 8 fixtures (230 stderr lines, 8,812 XML bytes).

@robobun

robobun commented Oct 3, 2026

Copy link
Copy Markdown
Collaborator Author

Closing: one PR in place of the stack. This refactor is optional and changes no output. The branch robobun/3eab222d/test-fail-file stays. I can open it against main after #44435 merges if a maintainer wants the private counters.

@robobun robobun closed this Oct 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants