fix(benchmark): render fork benchmark comments without dashboard uploads#2485
fix(benchmark): render fork benchmark comments without dashboard uploads#2485NathanDrake2406 wants to merge 4 commits into
Conversation
Fork pull request benchmark publishing treats a missing dashboard ingest secret as a skipped upload, but the PR comment formatter still required the dashboard response file. That makes fork CI fail after validation whenever a PR benchmark run has paired samples but no COMPAT_INGEST_SECRET. Write an explicit skipped upload response when the secret is unavailable, and render the PR comment from the validated paired benchmark artifact when dashboard comparison data is absent. This keeps upstream dashboard comments unchanged while allowing fork PRs to post local benchmark comparisons.
Fork benchmark publisher failures happen when the trusted publisher tries to render a PR comment after the dashboard upload skipped without producing performance-upload.json. Update the formatter regression test to omit the upload response file entirely so it covers that ENOENT path directly.
commit: |
Performance benchmarksCompared 0 improved · 0 regressed · 6 within ±1.5%
View detailed results and traces 🟢 improvement · 🔴 regression · ⚫ change below 1.5% · paired base/head |
|
/bigbonk review for issues |
There was a problem hiding this comment.
Reviewed the fork-benchmark-comment change. The core logic is sound and well-tested: the uploader now writes an explicit skipped-upload marker, and the formatter falls back to localComparison(resultBenchmarks) when no dashboard comparison is present. I verified the two new tests pass and traced the workflow wiring in .github/workflows/perf-publish.yml (the response path always exists in CI, and readUploadResponse's ENOENT branch is a reasonable extra safety net).
The upstream dashboard path is preserved: when uploadedComparison is truthy, comparisonSource === uploadedComparison, so line 36's comparisonSource.baseline and line 37's measurement mapping behave exactly as before, and dashboardUrl still links to the dashboard.
A few minor, non-blocking observations below. None block merge.
|
Review posted successfully. Summary of my findings on PR #2485: Verdict: Approve with minor notes — none blocking. The change correctly solves the fork-PR problem: the uploader writes an explicit
Three minor observations posted as inline comments:
|
Overview
perf-results.json.benchmarks/perf/upload-results.mjs,benchmarks/perf/format-pr-comment.mjs,tests/performance-benchmarks.test.tsWhy
The trusted publisher validates benchmark artifacts before publishing, but fork repositories do not have
COMPAT_INGEST_SECRET. That missing secret is a valid capability boundary, not a benchmark failure. The publisher should skip dashboard upload while still using the paired samples already present in the validated PR artifact.{ "uploaded": false, "reason": "missing_secret" }instead of silently omitting the response file.baselineSamplesandsampleswhen no dashboard comparison is available.comparison, the existing dashboard comparison and link are still used.What changed
COMPAT_INGEST_SECRETperformance-upload.jsonwas absent, andformat-pr-comment.mjsfailed withENOENT.ENOENTas a skipped upload and falls back to local paired comparison data.Validation
vp test run --project unit tests/performance-benchmarks.test.tsvp check tests/performance-benchmarks.test.ts benchmarks/perf/format-pr-comment.mjs benchmarks/perf/upload-results.mjsgit diff --checkReferences
performance-upload.json.main.