Skip to content

inspector: capture test line numbers for retroactive TestReporter.found events - #36882

Closed
robobun wants to merge 3 commits into
mainfrom
claude/farm/8e36155d/inspector-testreporter-retroactive-line
Closed

robobun wants to merge 3 commits into
mainfrom
claude/farm/8e36155d/inspector-testreporter-retroactive-line

Conversation

@robobun

@robobun robobun commented Aug 4, 2026 •

Copy link
Copy Markdown
Collaborator

Problem

When a debugger client sends TestReporter.enable after test collection has already started, bun test walks the already-discovered scope tree and emits a TestReporter.found event for each test. Every one of those events carried line: 0 (and no scriptId), so IDE test explorers placed them at the top of the file.

The same tests carry real line numbers when TestReporter.enable is sent before Inspector.initialized, and also on the late path if --reporter=junit happens to be on.

enable early:         describe:group line=2, test:one line=3, test:two line=4, ...
enable late:          describe:group line=0, test:one line=0, test:two line=0, ...
enable late + junit:  describe:group line=2, test:one line=3, test:two line=4, ...

Cause

The live path (InspectorTestReporterAgent::reportTestFound) derives the source location directly from the live CallFrame. The retroactive path has no call frame, so it reads BaseScope::line_no, which is populated by capture_test_line_number in src/runtime/test_runner/jest.rs. That function only walked the stack when test_options.reporters.junit was set; line capture was originally added for the JUnit reporter and the inspector path inherited the zero.

Fix

Also capture the line number at test registration time when an inspector is attached (vm.debugger.is_some()), so the retroactive walker has a real source location to emit. The common case (no inspector, no junit) still skips the stack walk.

Verification

Extended the existing retroactive-enable test in test/cli/inspect/test-reporter.test.ts to assert that every retroactively reported TestReporter.found event has line > 0 and the correct url, with an exact-line spot check.

# before fix
error: expect(received).toBeGreaterThan(expected)
Expected: > 0
Received: 0

# after fix
(pass) TestReporter inspector protocol > retroactively reports tests when TestReporter.enable is called after tests are discovered
(pass) TestReporter inspector protocol > assigns unique test IDs when TestReporter.enable lands mid-collection

[stamp-90s] gate passed · iteration 0 · 3 files touched

fails on main (without fix)
ASAN without fix: 1 FAILED
$ BUN_DEBUG_QUIET_LOGS=1 bun scripts/build.ts --profile=debug --quiet test "--reporter=junit" "--reporter-outfile=/tmp/mechgate.xml" test/cli/inspect/test-reporter.test.ts
bun test v1.4.0 (7498925f2)

test/cli/inspect/test-reporter.test.ts:
300 |     // as tests reported live during collection. The retroactive path reads the
301 |     // stored line number, which previously was only captured for the JUnit
302 |     // reporter and otherwise left at 0.
303 |     for (const t of testsArray) {
304 |       expect(t.url).toEndWith("delayed.test.ts");
305 |       expect(t.line).toBeGreaterThan(0);
                           ^
error: expect(received).toBeGreaterThan(expected)

Expected: > 0
Received: 0

      at <anonymous> (/workspace/bun/test/cli/inspect/test-reporter.test.ts:305:22)
(fail) TestReporter inspector protocol > retroactively reports tests when TestReporter.enable is called after tests are discovered [1832.08ms]
(pass) TestReporter inspector protocol > assigns unique test IDs when TestReporter.enable lands mid-collection [1531.59ms]

 1 pass
 1 fail
 19 expect() calls
Ran 2 tests across 1 file. [5.64s]
error: script "bd" exited with code 1
__F:1:S:0

release without fix: 2 FAILED
bun test v1.4.0-canary.1 (1498d7b77)

test/cli/inspect/test-reporter.test.ts:
300 |     // as tests reported live during collection. The retroactive path reads the
301 |     // stored line number, which previously was only captured for the JUnit
302 |     // reporter and otherwise left at 0.
303 |     for (const t of testsArray) {
304 |       expect(t.url).toEndWith("delayed.test.ts");
305 |       expect(t.line).toBeGreaterThan(0);
                           ^
error: expect(received).toBeGreaterThan(expected)

Expected: > 0
Received: 0

      at <anonymous> (/workspace/bun/test/cli/inspect/test-reporter.test.ts:305:22)
(fail) TestReporter inspector protocol > retroactively reports tests when TestReporter.enable is called after tests are discovered [35.51ms]
killed 1 dangling process
(fail) TestReporter inspector protocol > assigns unique test IDs when TestReporter.enable lands mid-collection [5000.21ms]
  ^ this test timed out after 5000ms.

 0 pass
 2 fail
 7 expect() calls
Ran 2 tests across 1 file. [5.18s]
__F:2:S:0
passes on PR (with fix)
ASAN with fix: all passed
$ BUN_DEBUG_QUIET_LOGS=1 bun scripts/build.ts --profile=debug --quiet test "--reporter=junit" "--reporter-outfile=/tmp/mechgate.xml" test/cli/inspect/test-reporter.test.ts
bun test v1.4.0 (7498925f2)

test/cli/inspect/test-reporter.test.ts:
(pass) TestReporter inspector protocol > retroactively reports tests when TestReporter.enable is called after tests are discovered [1604.40ms]
(pass) TestReporter inspector protocol > assigns unique test IDs when TestReporter.enable lands mid-collection [1536.59ms]

 2 pass
 0 fail
 30 expect() calls
Ran 2 tests across 1 file. [5.18s]
__F:0:S:0

release with fix: all passed
$ bun scripts/build.ts --profile=release
[configured] bun-profile → bun (stripped) in 700ms (unchanged)
ninja: Entering directory `/workspace/bun/build/release'
[1/6] gen generated_host_exports.rs
generated_host_exports.rs: 94 exports (host=3, lazy=10, generic=81, rust=0); 240 extern-C blocks audited
[1/6] cargo bun_bin → libbun_rust.a (--target x86_64-unknown-linux-gnu)

  nightly-2026-07-20-x86_64-unknown-linux-gnu unchanged - rustc 1.99.0-nightly (9f36de775 2026-07-19)

�[1m�[92m   Compiling�[0m bun_core v0.0.0 (/workspace/bun/src/bun_core)
�[1m�[92m   Compiling�[0m bun_errno v0.0.0 (/workspace/bun/src/errno)
�[1m�[92m   Compiling�[0m bun_ptr v0.0.0 (/workspace/bun/src/ptr)
�[1m�[92m   Compiling�[0m bun_boringssl_sys v0.0.0 (/workspace/bun/src/boringssl_sys)
�[1m�[92m   Compiling�[0m bun_safety v0.0.0 (/workspace/bun/src/safety)
�[1m�[92m   Compiling�[0m bun_zlib_sys v0.0.0 (/workspace/bun/src/zlib_sys)
�[1m�[92m   Compiling�[0m bun_cares_sys v0.0.0 (/workspace/bun/src/cares_sys)
�[1m�[92m   Compiling�[0m bun_zstd v0.0.0 (/workspace/bun/src/zstd)
�[1m�[92m   Compiling�[0m bun_picohttp v0.0.0 (/workspace/bun/src/picohttp)
�[1m�[92m   Compiling�[0m bun_brotli v
... (truncated)
diff hotspot
src/runtime/test_runner/bun_test.rs    |  2 +-
 src/runtime/test_runner/jest.rs        |  2 +-
 test/cli/inspect/test-reporter.test.ts | 14 ++++++++++++++
 3 files changed, 16 insertions(+), 2 deletions(-)

gate history · 1 passed · 0 rejected · iteration 0

evidence per changed file
file                                    reads  edits  tests
src/runtime/test_runner/bun_test.rs         1      1      0
src/runtime/test_runner/jest.rs             3      3      0
test/cli/inspect/test-reporter.test.ts      1      1      0

…nd events

When TestReporter.enable arrives after test collection has started, the
retroactive walker emits TestReporter.found for each already-discovered
test using the line number stored on BaseScope. That field was only
populated when the JUnit reporter was enabled, so retroactively reported
tests carried line: 0 (and so landed at the top of the file in IDE test
explorers) unless --reporter=junit happened to be on.

Capture the line number at registration time whenever an inspector is
attached as well, so the retroactive path has a real source location to
report.
@robobun

robobun commented Aug 4, 2026 •

Copy link
Copy Markdown
Collaborator Author

Status: diff is green. test/cli/inspect/test-reporter.test.ts passed on every lane in both builds (#88775, #88786).

Remaining CI failures are all tagged [flaky] and unrelated to this change (napi GC timing on Windows, bun install registry/security-scanner, fastutf8stream-reopen, zlib estimated-size GC). The worker-transfer-terminate-stress asan assertion that appeared in #88775 did not recur in #88786.

Ready for review.

@github-actions github-actions Bot added the claude label Aug 4, 2026
@coderabbitai

coderabbitai Bot commented Aug 4, 2026 •

Copy link
Copy Markdown
Contributor

Review Change Stack

Walkthrough

The test runner now captures test line numbers when inspector mode is enabled. The inspector reporter regression test verifies source URLs and line numbers for retroactively reported tests.

Changes

Inspector test location reporting

Layer / File(s) Summary
Capture inspector test locations
src/runtime/test_runner/bun_test.rs, src/runtime/test_runner/jest.rs
The test runner captures debugger identifiers and call-frame line numbers when inspector mode is enabled.
Validate retroactive reporting
test/cli/inspect/test-reporter.test.ts
The regression test verifies source URLs and positive line numbers, including the exact line for test A1.

Possibly related PRs

Suggested reviewers: jarred-sumner

🚥 Pre-merge checks | ✅ 4
✅ Passed checks (4 passed)
Check name Status Explanation
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Title check ✅ Passed The title clearly describes the main change: capturing test line numbers for retroactive inspector TestReporter.found events.
Description check ✅ Passed The description explains the problem, cause, fix, and verification with relevant technical details, although it does not use the template headings.

Comment @coderabbitai help to get the list of available commands.

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM — one-line condition widening so capture_test_line_number also runs when the inspector is attached, fixing line: 0 in retroactive TestReporter.found events.

What was reviewed:

  • Verified is_inspector_enabled() reads vm.debugger.is_some(), which is set for --inspect-wait before collection runs.
  • Confirmed the retroactive walk in jsc_hooks.rs:1759,1786 reads base.line_no, so populating it is the right layer.
  • Checked the exact-line assertion (testA1?.line === 6) against the fixture's template-literal line numbering.
  • No-inspector/no-junit path still skips the stack walk (perf-neutral for the common case).
Extended reasoning...

Overview

This PR widens a single condition in capture_test_line_number (src/runtime/test_runner/jest.rs:821) from reporters.junit to reporters.junit || vm.is_inspector_enabled(), so that test/describe registration captures the source line number whenever an inspector is attached — not only when the JUnit reporter is on. It also updates the doc comment on BaseScope::line_no to match, and extends the existing retroactive-enable test in test/cli/inspect/test-reporter.test.ts to assert that every retroactively reported TestReporter.found event carries a real url and line > 0, plus an exact-line spot check.

Security risks

None. This is a debugger/inspector metadata path; the added call is a boolean read (vm.debugger.is_some()) and a stack-frame line-number lookup that already ran under --reporter=junit.

Level of scrutiny

Low. The functional change is one boolean disjunct in a gate that already existed. I verified:

  • is_inspector_enabled() at src/jsc/VirtualMachine.rs:984 returns self.debugger.is_some(), which is populated when --inspect/--inspect-wait initializes the debugger — so it is true during collection in the fixture.
  • The consumer (retroactively_report_scope in src/runtime/jsc_hooks.rs:1753,1780) reads base.line_no and passes it to report_test_found_with_location, so populating line_no is the correct fix layer.
  • The common case (bun test with no inspector, no junit) still short-circuits before the FFI stack walk, so there is no perf regression on the hot path.
  • The Jest::runner() borrow followed by global_this.bun_vm() is the pre-existing pattern here; the new call only reads a disjoint VM field and does not touch TestRunner.

Other factors

The test extension is added alongside (not replacing) existing assertions in the retroactive-enable test, and includes both a broad invariant (line > 0 for all 5 entries, correct url) and an exact-line spot check to catch off-by-one regressions. The PR body documents fail-before/pass-after with released vs. debug bun. The comment update on BaseScope::line_no keeps the field doc accurate.

Comment thread src/runtime/test_runner/jest.rs Resolved

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM — one-line gate widening with clear fail-before/pass-after evidence.

What was reviewed:

  • Confirmed is_inspector_enabled() is debugger.is_some(), which is set before collection under --inspect-wait/--inspect, so line_no is captured for the retroactive walk in jsc_hooks.rs:1759/1786.
  • Checked the gate is intentionally broader than the sibling test_id_for_debugger gate (agent.is_enabled()) — that's the point: capture must happen before TestReporter.enable lands.
  • Verified the exact-line assertion (test A1 on line 6) against the fixture's leading newline.
Extended reasoning...

Overview

Two-line runtime change plus test assertions. capture_test_line_number in src/runtime/test_runner/jest.rs previously only walked the stack for a source line when --reporter=junit was on; it now also does so when vm.debugger.is_some(). The doc comment on BaseScope::line_no in bun_test.rs is updated to match. The existing retroactive-enable test in test/cli/inspect/test-reporter.test.ts gains assertions that every retroactively reported TestReporter.found event carries url ending in the fixture filename and line > 0, plus an exact-line spot check.

Security risks

None. This only affects when a stack-walk for a source line number is performed during test()/describe() registration, gated on the presence of a debugger. No user-controlled input, no new FFI surface.

Level of scrutiny

Low. The runtime change is a one-token OR added to an existing conditional; the extra work (Bun__CallFrame__getLineNumber) is the same call the JUnit path already exercises, and it only fires under an attached inspector — a debugging scenario where the extra stack walk per test registration is negligible. I traced the retroactive consumer in src/runtime/jsc_hooks.rs:retroactively_report_scope and confirmed it reads base.line_no for both describe and test entries, so the fix lands at the layer that owns the invariant.

Other factors

  • The gate correctly uses debugger.is_some() rather than test_reporter_agent.is_enabled(): the whole bug is that the agent enables after collection, so line capture must be unconditional on debugger presence.
  • Evidence in the PR body shows fail-before (line 0 received) on both ASAN debug and release without the fix, pass-after on both with it. The inner spawned subprocess does not pass --reporter=junit, so the fail-before is genuine.
  • Test additions extend the existing retroactive test in place (no new file), assert on already-collected event data, and include an exact-line check whose arithmetic I verified against the template-literal fixture.
  • The comment-cop bot's concern about a paragraph comment was addressed in 7498925 and the thread is resolved.

@github-actions

github-actions Bot commented Nov 3, 2026

Copy link
Copy Markdown
Contributor

Closing this PR because it has been inactive for more than 90 days.

@robobun

robobun commented Nov 3, 2026

Copy link
Copy Markdown
Collaborator Author

The stale workflow closed this with no maintainer review. The work continues in #47868.

The gate here (vm.is_inspector_enabled()) is also true for BUN_INSPECT_CONNECT_TO, which the VS Code extension sets in every terminal. It would add a stack walk per test() to every bun test run from there. #47868 decides the lookup once at startup and leaves that variable out. It also gives the early found, the late found and JUnit one stored line.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants