Support vitest 5 by GuillaumeLagrange · Pull Request #86 · CodSpeedHQ/codspeed-node

GuillaumeLagrange · 2026-06-29T07:59:11Z

No description provided.

sheremet-va · 2026-06-30T09:42:07Z

+  }
+}
+
+function patchRunBenchmarks(): void {


By the way, you can do that just in a setup file without overriding the runner. This should also make it support the browser(?)

If there is any feedback, I would love to address them on vitest side; we would also like to use codspeed after Vitest 5 is out in our own repo

@sheremet-va thanks for the heads up, currently trying out this approach.

The main pain point I am running into is that we are struggling to highjack the benchmark runner when we are in simulation/analysis mode.

When in this mode, what we used to do is highjack the whole run function, to have a very simple sequence of

warmup

manually call optimizer

start valgrind instrumentation

execute code once

stop

But I'm struggling a bit to find a good way to replicate the behavior without overriding the whole runner.
I still do not have a proper api request, nor am I sure it's the way to go for the analysis mode, becuase maybe running browser mode under valgrind is not a good idea in the first place, and the main motivation for not highjacking the whole runner is actually to be as compatible as possible with the browser mode.

Could it be possible to have a way for the plugin to hook/customize the createTinybench function? Maybe by defining some sort of abstract "benchmark backend" interface that we could fullfill, that would default to a tinybench implementation. It would also make it easier to switch away from tinybench on your end at some point.

The way tinybench is instanciated/ran does not allow for much flexibility on our end, and I cannot find a good implementation without overriding the whole runner as we did before for our analysis mode.

I'd love to exchange over this over discord/slack/a call if you want, or we can keep things here on github.

Feel free to join the discord - https://chat.vitest.dev/

I will give you the ecosystem role, we can discuss it there. I just need your nickname

I've joined and pinged on via MP, my nickname on the server is the same is github

codspeed · 2026-07-01T09:07:39Z

Merging this PR will regress 4 benchmarks

⚠️

Unknown Walltime execution environment detected

Using the Walltime instrument on standard Hosted Runners will lead to inconsistent data.

For the most accurate results, we recommend using CodSpeed Macro Runners: bare-metal machines fine-tuned for performance measurement consistency.

⚠️

Different runtime environments detected

Some benchmarks with significant performance changes were compared across different runtime environments,
which may affect the accuracy of the results.

Open the report in CodSpeed to investigate

⚡ 4 improved benchmarks
❌ 4 regressed benchmarks
✅ 177 untouched benchmarks
🆕 79 new benchmarks
⏩ 53 skipped benchmarks¹

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

	Mode	Benchmark	`BASE`	`HEAD`	Efficiency
❌	Simulation	`switch 1`	204.5 µs	689.9 µs	-70.37%
❌	Memory	`wait 500ms`	15.5 KB	21.5 KB	-27.99%
❌	Simulation	`iterative fibo 20`	26.9 µs	30.7 µs	-12.36%
❌	WallTime	`test_recursive_cached_fibo_10`	2.3 µs	2.6 µs	-12.09%
⚡	Simulation	`recursive fibo 10`	1,372.1 µs	299.9 µs	×4.6
⚡	Simulation	`recursive fibo 15`	636.9 µs	352.1 µs	+80.88%
⚡	WallTime	`switch 1`	84 ns	72 ns	+16.67%
⚡	WallTime	`test_iterative_fibo_10`	120 ns	108 ns	+11.11%
🆕	WallTime	`iterative fibo 15`	N/A	672 ns	N/A
🆕	WallTime	`iterative fibo 20`	N/A	672 ns	N/A
🆕	WallTime	`recursive fibo 15`	N/A	20.4 µs	N/A
🆕	WallTime	`recursive fibo 20`	N/A	219.2 µs	N/A
🆕	WallTime	`iterative`	N/A	552 ns	N/A
🆕	WallTime	`recursive`	N/A	20.2 µs	N/A
🆕	WallTime	`iterative`	N/A	564 ns	N/A
🆕	WallTime	`recursive`	N/A	218.9 µs	N/A
🆕	WallTime	`fibo 10`	N/A	2.3 µs	N/A
🆕	WallTime	`fibo 15`	N/A	20.3 µs	N/A
🆕	WallTime	`long body`	N/A	267.2 µs	N/A
🆕	WallTime	`short body`	N/A	1.5 µs	N/A
...	...	...	...	...	...

ℹ️ Only the first 20 benchmarks are displayed. Go to the app to view all benchmarks.

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.

_{Comparing cod-2931-prepare-compatibility-with-vitest-5 (9007e6e) with main (3258048)}

53 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

greptile-apps · 2026-07-19T18:51:01Z

Greptile Summary

This PR adds Vitest 5 support to the CodSpeed Vitest plugin. The main changes are:

A new Vitest 5 benchmark provider path.
Legacy Vitest 3/4 runner code moved under src/legacy.
Shared instrumentation and walltime result conversion helpers.
Updated Vitest dependency ranges and a Vitest 4 example project.

Confidence Score: 4/5

The new Vitest 5 path can record distinct benchmarks under one CodSpeed URI, and legacy fallback can skip instrumentation.

Multiple bench() registrations in one Vitest 5 group can share one CodSpeed benchmark identity.
Unknown Vitest version detection can route supported Vitest 3/4 runs through the v5 gating path.
The remaining changes are mostly layout, tests, examples, and shared helper refactors.

packages/vitest-plugin/src/v5/provider.ts and packages/vitest-plugin/src/vitestBackend.ts

Important Files Changed

Filename	Overview
packages/vitest-plugin/src/v5/provider.ts	Adds the Vitest 5 provider, but the current URI construction can collapse multiple registrations in one group.
packages/vitest-plugin/src/vitestBackend.ts	Adds version-specific backend selection, with a risky unknown-version fallback to the v5 path.
packages/vitest-plugin/src/instrument.ts	Adds shared analysis and walltime instrumentation helpers.
packages/vitest-plugin/src/index.ts	Delegates plugin apply and config behavior to the selected Vitest backend.
packages/vitest-plugin/src/legacy/walltime.ts	Moves the legacy walltime runner to the new legacy layout.
packages/vitest-plugin/src/globalSetup.ts	Makes setup and teardown idempotent for repeated Vitest 5 setup calls.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[Vitest run] --> B{Detected Vitest major}
  B -->|3 or 4| C[Legacy backend]
  C --> D[legacy analysis or walltime runner]
  B -->|5 or newer| E[V5 backend]
  E --> F{benchmark enabled}
  F -->|no| G[default Vitest behavior]
  F -->|yes| H[v5 benchmark provider]
  H --> I{CodSpeed mode}
  I -->|analysis or simulation| J[run one measured pass]
  I -->|walltime| K[run tinybench with hooks]
  J --> L[CodSpeed benchmark result]
  K --> L

%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
flowchart TD
  A[Vitest run] --> B{Detected Vitest major}
  B -->|3 or 4| C[Legacy backend]
  C --> D[legacy analysis or walltime runner]
  B -->|5 or newer| E[V5 backend]
  E --> F{benchmark enabled}
  F -->|no| G[default Vitest behavior]
  F -->|yes| H[v5 benchmark provider]
  H --> I{CodSpeed mode}
  I -->|analysis or simulation| J[run one measured pass]
  I -->|walltime| K[run tinybench with hooks]
  J --> L[CodSpeed benchmark result]
  K --> L

Comments Outside Diff (2)

packages/vitest-plugin/src/v5/provider.ts, line 1778 (link)

Group URI Drops Benchmark Name

When a Vitest 5 test registers multiple bench(name, ...) entries, this provider builds one URI from the group and reuses it for every registration. The legacy paths append the benchmark task name, so the new path can attribute distinct measurements to the same CodSpeed benchmark and merge or overwrite their results.

Prompt To Fix With AI

This is a comment left during a code review.
Path: packages/vitest-plugin/src/v5/provider.ts
Line: 1778

Comment:
**Group URI Drops Benchmark Name**

When a Vitest 5 test registers multiple `bench(name, ...)` entries, this provider builds one URI from the group and reuses it for every registration. The legacy paths append the benchmark task name, so the new path can attribute distinct measurements to the same CodSpeed benchmark and merge or overwrite their results.

How can I resolve this? If you propose a fix, please make it concise.

packages/vitest-plugin/src/vitestBackend.ts, line 1910-1911 (link)

Unknown Version Selects V5

When Vitest resolution from process.cwd() fails for a supported Vitest 3/4 project, this fallback selects the v5 backend. A v3/v4 benchmark run then has no test.benchmark.enabled flag, so config() returns undefined and the legacy runner is never installed, leaving CodSpeed runs uninstrumented without an error.

Prompt To Fix With AI

This is a comment left during a code review.
Path: packages/vitest-plugin/src/vitestBackend.ts
Line: 1910-1911

Comment:
**Unknown Version Selects V5**

When Vitest resolution from `process.cwd()` fails for a supported Vitest 3/4 project, this fallback selects the v5 backend. A v3/v4 benchmark run then has no `test.benchmark.enabled` flag, so `config()` returns `undefined` and the legacy runner is never installed, leaving CodSpeed runs uninstrumented without an error.

How can I resolve this? If you propose a fix, please make it concise.

Prompt To Fix All With AI

Fix the following 3 code review issues. Work through them one at a time, proposing concise fixes.

---

### Issue 1 of 3
packages/vitest-plugin/src/v5/provider.ts:1778
**Group URI Drops Benchmark Name**

When a Vitest 5 test registers multiple `bench(name, ...)` entries, this provider builds one URI from the group and reuses it for every registration. The legacy paths append the benchmark task name, so the new path can attribute distinct measurements to the same CodSpeed benchmark and merge or overwrite their results.

### Issue 2 of 3
packages/vitest-plugin/src/vitestBackend.ts:1910-1911
**Unknown Version Selects V5**

When Vitest resolution from `process.cwd()` fails for a supported Vitest 3/4 project, this fallback selects the v5 backend. A v3/v4 benchmark run then has no `test.benchmark.enabled` flag, so `config()` returns `undefined` and the legacy runner is never installed, leaving CodSpeed runs uninstrumented without an error.

### Issue 3 of 3
packages/vitest-plugin/src/globalSetup.ts:15
**Untracked TODO Comment**

This new TODO has no Linear issue reference, which violates the repository rule for TODO comments. Please change it to the `TODO(COD-XXX): ...` form or remove it.

```suggestion
// TODO(COD-XXX): Check if this can be avoided
```

_{Reviews (1): Last reviewed commit: "feat(vitest-plugin): use the vitest 5 be..." | Re-trigger Greptile}

greptile-apps · 2026-07-19T18:51:06Z

+let setupHappened = false;
 let teardownHappened = false;

+// TODO: Check if this can be avoided


Untracked TODO Comment

This new TODO has no Linear issue reference, which violates the repository rule for TODO comments. Please change it to the TODO(COD-XXX): ... form or remove it.

Suggested change

// TODO: Check if this can be avoided

// TODO(COD-XXX): Check if this can be avoided

Rule Used: Every TODO comment must reference a Linear issue: ... (source)

Prompt To Fix With AI

This is a comment left during a code review. Path: packages/vitest-plugin/src/globalSetup.ts Line: 15 Comment: **Untracked TODO Comment** This new TODO has no Linear issue reference, which violates the repository rule for TODO comments. Please change it to the `TODO(COD-XXX): ...` form or remove it. ```suggestion // TODO(COD-XXX): Check if this can be avoided ``` **Rule Used:** Every TODO comment must reference a Linear issue: ... ([source](https://app.greptile.com/codspeed/-/custom-context?memory=65193bc9-f65b-477d-9521-104b5aac5931)) How can I resolve this? If you propose a fix, please make it concise.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

florian-lefebvre · 2026-09-22T10:13:35Z

Hi @GuillaumeLagrange, what's the status of this PR? We would need it for Astro

GuillaumeLagrange · 2026-09-28T08:12:28Z

Hi @GuillaumeLagrange, what's the status of this PR? We would need it for Astro

Hey! The support for the benchmark provider has been merged and released upstream by vitest, we plan to merge and release this PR early this week! Don't hesitate to continue pinging me to get updates, and sorry about the delay

A plugin behaves differently depending on whether CodSpeed drives the run, and each mode takes a different path through the plugin, so all three have to be run when developing or reviewing one. Write down the commands and the expected output per mode, and ignore the local `.codspeed` result directory those runs produce. Co-Authored-By: Claude <noreply@anthropic.com>

Vitest 5 removed the dedicated `NodeBenchmarkRunner` and the `vitest/runners` / `vitest/suite` entrypoints. Benchmarks are now declared through a `bench` test-context fixture and executed by a `benchmark.provider`: a module Vitest hands the registered functions, their options and the declaring test, and whose results it treats as authoritative. Detect the installed Vitest generation and select the integration behind a `VitestBackend` abstraction so the rest of the plugin never inspects the version: - v3/4 keep the custom benchmark runner per instrument mode. Their Vitest lookups move behind `legacy/compat`, which owns the types of the removed benchmark backend so the package type-checks against Vitest 5. - v5 registers CodSpeed as the benchmark provider, which runs the analysis and walltime modes off the registrations it receives. On v5, benchmark URIs carry the registration name after the test path, so a test declaring several benchmarks through `bench.compare()` reports one benchmark per registration. Gate the injection on the instrument mode: Vitest 5 clones its benchmark project after the Vite config hooks ran, so a benchmark run can no longer be detected from the incoming config, and CodSpeed only ever drives benchmark runs. Without CodSpeed the plugin injects nothing and Vitest runs the benchmarks through its own tinybench provider. Vitest 5 also runs `globalSetup` once per project, including the cloned benchmark project, so make setup and teardown idempotent instead of failing the run on the second teardown. Bump the plugin's own dev dependency to Vitest 5 and move its benches to the fixture API, reading imported bindings into a local before the measured loop so the module-runner getters stay out of the samples. Refs COD-2931 Co-Authored-By: Claude <noreply@anthropic.com>

Now that the plugin's own dev dependency tracks Vitest 5, add a dedicated Vitest 4 example so the legacy (v3/4) benchmark seam keeps explicit coverage alongside the existing with-vitest-v3 example. Mirrors that example, pinning vitest ^4.1.9. Refs COD-2931 Co-Authored-By: Claude <noreply@anthropic.com>

Benchmark the plugin against Vitest 5 in CI the way the v3 and v4 examples do for their majors, using the `bench` fixture and `bench.compare()`. Refs COD-2931 Co-Authored-By: Claude <noreply@anthropic.com>

Benchmarks are declared through the `bench` test-context fixture on Vitest 5; keep the top-level `bench()` form as a note for Vitest 3 and 4. Drop the fallback log line from the sample output, which the plugin no longer prints. Refs COD-2931 Co-Authored-By: Claude <noreply@anthropic.com>

GuillaumeLagrange changed the title ~~Cod 2931 prepare compatibility with vitest 5~~ Support vitest 5 Jun 29, 2026

sheremet-va reviewed Jun 30, 2026

View reviewed changes

GuillaumeLagrange force-pushed the cod-2931-prepare-compatibility-with-vitest-5 branch 5 times, most recently from 25ef927 to e04f0da Compare July 1, 2026 08:59

GuillaumeLagrange force-pushed the cod-2931-prepare-compatibility-with-vitest-5 branch 2 times, most recently from f511dee to c3a356b Compare July 1, 2026 12:57

greptile-apps Bot reviewed Jul 19, 2026

View reviewed changes

HiDeoo mentioned this pull request Sep 3, 2026

Update to Vitest 5 withastro/starlight#4180

Draft

2 of 3 tasks

GuillaumeLagrange force-pushed the cod-2931-prepare-compatibility-with-vitest-5 branch from eec7ec7 to eb2042e Compare September 4, 2026 16:46

delucis mentioned this pull request Sep 24, 2026

chore: add benchmarks withastro/starlight#4168

Draft

This was referenced Sep 24, 2026

Add CodSpeed continuous performance benchmarks ITISFoundation/mmux_vite#623

Closed

EPIC: CI, dependency, coverage, and performance modernization ITISFoundation/mmux_vite#624

Open

GuillaumeLagrange force-pushed the cod-2931-prepare-compatibility-with-vitest-5 branch from eb2042e to d6abe82 Compare September 30, 2026 12:29

GuillaumeLagrange and others added 4 commits September 30, 2026 14:52

test(vitest-plugin): add with-vitest-v5 example

2a2f565

Benchmark the plugin against Vitest 5 in CI the way the v3 and v4 examples do for their majors, using the `bench` fixture and `bench.compare()`. Refs COD-2931 Co-Authored-By: Claude <noreply@anthropic.com>

GuillaumeLagrange force-pushed the cod-2931-prepare-compatibility-with-vitest-5 branch from d6abe82 to 9007e6e Compare September 30, 2026 12:58

Navigation Menu

Support vitest 5 - #86

Support vitest 5#86
GuillaumeLagrange wants to merge 5 commits into
mainfrom
cod-2931-prepare-compatibility-with-vitest-5

GuillaumeLagrange commented Jun 29, 2026

Uh oh!

sheremet-va Jun 30, 2026

Uh oh!

GuillaumeLagrange Jul 1, 2026

Uh oh!

GuillaumeLagrange Jul 1, 2026

Uh oh!

sheremet-va Jul 1, 2026

Uh oh!

GuillaumeLagrange Jul 1, 2026

Uh oh!

codspeed Bot commented Jul 1, 2026 •

edited

Loading

Uh oh!

greptile-apps Bot commented Jul 19, 2026 •

edited

Loading

Flowchart

Comments Outside Diff (2)

Uh oh!

greptile-apps Bot Jul 19, 2026

Uh oh!

florian-lefebvre commented Sep 22, 2026

Uh oh!

GuillaumeLagrange commented Sep 28, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

	// TODO: Check if this can be avoided
	// TODO(COD-XXX): Check if this can be avoided

Conversation

GuillaumeLagrange commented Jun 29, 2026

Uh oh!

sheremet-va Jun 30, 2026

Choose a reason for hiding this comment

Uh oh!

GuillaumeLagrange Jul 1, 2026

Choose a reason for hiding this comment

Uh oh!

GuillaumeLagrange Jul 1, 2026

Choose a reason for hiding this comment

Uh oh!

sheremet-va Jul 1, 2026

Choose a reason for hiding this comment

Uh oh!

GuillaumeLagrange Jul 1, 2026

Choose a reason for hiding this comment

Uh oh!

codspeed Bot commented Jul 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merging this PR will regress 4 benchmarks

Performance Changes

Footnotes

Uh oh!

greptile-apps Bot commented Jul 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Greptile Summary

Confidence Score: 4/5

Important Files Changed

Flowchart

Comments Outside Diff (2)

Uh oh!

greptile-apps Bot Jul 19, 2026

Choose a reason for hiding this comment

Uh oh!

florian-lefebvre commented Sep 22, 2026

Uh oh!

GuillaumeLagrange commented Sep 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

codspeed Bot commented Jul 1, 2026 •

edited

Loading

greptile-apps Bot commented Jul 19, 2026 •

edited

Loading

GuillaumeLagrange commented Sep 28, 2026 •

edited

Loading