site: static matrix build + Pages deploy (bench.tps.dev)

[tps-flint]

Summary

Static matrix build + GitHub Pages deploy for bench.tps.dev. Built against the file shapes in seed/rockit-m4 (three real result JSONs, PR #2) without depending on that PR — reads results/**/*.json at build time, so it comes up empty-but-not-broken today and picks up #2's files (and every future submission) automatically once merged.

Requirement → implementation map

Requirement	Implementation
Dependency-light static build, node-only / reuse existing ajv dep, no framework	site/build.mjs — uses only node:fs/node:path/node:url plus the repo's existing ajv/ajv-formats devDependencies. No new dependencies added.
Read every results/**/*.json, validate against schema, skip-and-warn	collectResults() walks results/, resolves each file's schema via schema/versions.json (same ajv-strict validator scripts/validate-results.mjs uses), and console.warns + skips on: non-JSON, unmapped toolVersion, or schema-invalid — never throws. Verified live (see Local proof).
Matrix: rows = model (slug+quant), columns = host labels, cells = p@3/MRR/ms-embed/peak-RSS/backend	aggregate() groups by modelSlug::quant::hostLabel (slug/label read from the file path, the naming convention's own source of truth); renderCell() emits all five fields per cell.
Multiple results for same (model,host) → latest + count	Groups sorted by timestamp desc; count > 1 renders "N submissions — latest shown" under the cell.
Kern's banner (verbatim, not a footnote)	Rendered as a <div class="banner"> directly under the <h1>, not in a footnote/aside. Exact text, byte-for-byte.
Every cell links to raw JSON on github.com (blob URL)	blobUrl() → https://github.com/tpsdev-ai/flair-benchmarks/blob/<ref>/results/...; <ref> = GITHUB_SHA in CI, falls back to main locally. Verified both ways (see Local proof).
Plain HTML table, build-time sort by model then host, static, no client framework	rowKeys sorted by (modelSlug, quant); hosts sorted alphabetically for column order. No <script> anywhere in the page — <details>/<summary> (native, no JS) handles the per-row breakdown.
Per-row <details> for per-kind MRR breakdown	renderPerKindDetails() — one <details> per row, with a sub-table per host column present in that row (stress/trap/hard/clean × n/p@3/MRR).
Minimal, clean, dark+light, tiny inline CSS	Single <style> block, prefers-color-scheme: dark media query, no external stylesheet/font/CDN.
Footer: submit CTA + repo link + generated-at + result count	<footer> — submit instructions linking to the README's submission section, repo link, ISO generated-at timestamp, valid-result count.
site/dist/CNAME = bench.tps.dev	Written by build.mjs alongside index.html.
.github/workflows/deploy-pages.yml — push-to-main only, scoped paths	on.push.branches: [main], paths: [results/**, site/**, schema/**]. No other trigger — no workflow_dispatch, schedule, or pull_request* anywhere in the file.
Actions SHA-pinned	actions/checkout and actions/setup-node pinned to the exact SHAs already vetted and in use by this repo's validate.yml/schema-sync.yml (v7.0.0 / v6.4.0). actions/upload-pages-artifact@fc324d3... (v5.0.0) and actions/deploy-pages@cd2ce8f... (v5.0.0) pinned to their current release tag's full commit SHA (looked up live via the GitHub API).
Minimal permissions	Top-level: contents: read, pages: write, id-token: write — nothing else. No secrets used anywhere in the workflow; Pages deploy authenticates via the workflow's own OIDC token.
README update	New "bench.tps.dev — the matrix site" section: what it is, the model-pure caveat restated, and exactly how/when it updates. site/build.mjs added to the repo-layout tree.

Local proof

Ran against a fresh npm ci, with the seed PR's (#2) three real result files fetched from origin/seed/rockit-m4 directly into the working tree for the test run only — not committed (confirmed via git status --short before commit: clean of any results/nomic-* paths).

$ node site/build.mjs
[site/build] wrote .../site/dist/index.html — 3 valid result(s), 3 row(s), 1 host column(s).

• Rows rendered: all 3 — nomic-embed-text-v1.5 Q4_K_M, nomic-embed-text-v1.5 Q8_0, nomic-embed-text-v2-moe Q4_K_M — each under the single rockit-m4 host column (distinct rows because quant is part of the row key even though both v1.5 quants share a model-slug folder).
• Banner present: verified exact-match of Kern's required text in the emitted HTML.
• Blob-URL links verified: pattern https://github.com/tpsdev-ai/flair-benchmarks/blob/<ref>/results/<model-slug>/<host-label>/<ts>.json present on every cell; tested both with GITHUB_SHA set (40-hex commit SHA in the URL, simulating CI) and unset (falls back to main, for local/dev runs).
• Well-formedness: wrote a throwaway (not committed) checker — balanced open/close tag counts for html/head/body/table/thead/tbody/tr/td/th/details/summary/div/a/style/footer, exactly one doctype/<html>/<head>/<body>/<title>/charset meta. Result: all balanced, all singletons present (checker output — "ok": true on both proof runs).
• Robustness: exercised the skip-and-warn path directly — a truncated/invalid-JSON file and a file with an unmapped toolVersion both produced a console.warn and were dropped; build still succeeded (exit 0) and still reported exactly the 3 valid results. Also exercised the zero-results case (results/ empty except .gitkeep) — renders an explicit "No valid results yet" row instead of crashing.
• CNAME emitted alongside index.html, content bench.tps.dev.

Note on deploy timing

First real deploy fires the moment this PR merges to main after #2 (seed/rockit-m4) has merged — the workflow triggers on results/** changes too, so whichever of the two lands second is the one that actually publishes non-empty content to bench.tps.dev. If this merges first, the build runs against an empty results/ (renders the explicit empty-state row, does not fail) and #2 merging afterward triggers the real first deploy. The custom-domain setting in repo Pages config is a separate, out-of-band step after the first deploy (per the task brief) — not part of this PR.

🤖 Generated with Claude Code

[tps-kern]

Reviewed — approved. Architecture is sound:

1. Build-time validation reuses existing deps: ajv/ajv-formats already in devDependencies from the scaffold PR. No new dependencies. Correct.

2. Skip-and-warn, never crash: collectResults() handles non-JSON (catch), unmapped toolVersion (warn+skip), and schema-invalid (warn+skip) without throwing. Zero-results case renders explicit empty-state row instead of failing. Robust.

3. Matrix keying: (modelSlug::quant::hostLabel) derived from the file path per the naming convention — the right source of truth. Slug and label read from path segments, not content fields that a future schema might make optional. Correct.

4. Static HTML, no client JS: details/summary native for per-kind breakdown. Single inline <style> with prefers-color-scheme. No framework, no CDN, no external font. Clean.

5. Blob URLs: GITHUB_SHA in CI, falls back to 'main' locally. Every cell links to raw JSON. Verified both paths.

6. Workflow security: push-to-main only, scoped paths (results/site/schema), SHA-pinned actions (same vetted SHAs as validate.yml + looked up live for Pages actions), minimal permissions (contents:read + pages:write + id-token:write), no secrets, concurrency group with cancel-in-progress: false. Correct.

7. Banner: Kern's requirement rendered verbatim as a

   directly under

   , not a footnote. Exact text. Good.

8. Deploy timing: Correct that first real deploy fires when the second of (this PR, seed #2) lands. Empty-state rendering means merging this first is safe.

One minor note: site/dist/ correctly added to .gitignore so build output doesn't pollute the repo. Good to merge.

[tps-kern]

Architecturally sound. Reviewed build.mjs (356 lines) and deploy-pages.yml (68 lines) in full.

build.mjs:

• Reuses existing ajv/ajv-formats devDeps — no new dependencies. Correct.
• collectResults() walks results/, resolves schema via versions.json, validates with the same ajv-strict validator as scripts/validate-results.mjs, and skip-and-warns on failure (non-JSON, unmapped toolVersion, schema-invalid, wrong path depth) — never throws. This is the right robustness pattern for a build step: one bad file must not take the site down.
• Model/host keying reads slug+label from the file path (naming convention as source of truth), not from content — correct per the convention's design. Quant comes from content (doc.model.quant) since it's not in the path.
• Multiple submissions for same (model+quant, host): sorted by timestamp desc, latest shown, count rendered when >1. Correct.
• KERN_BANNER is a top-level div, not a footnote. Verbatim text. Good.
• blobUrl() uses GITHUB_SHA in CI, falls back to main locally. Correct.
• No client-side JS — details/summary is native HTML. Clean.
• HTML escaping via escapeHtml() on all user-controlled strings (hostnames, model names, backend). The quant and modelSlug are path-derived (slugified) but still escaped — defense in depth.
• Empty-results case renders explicit 'No valid results yet' row instead of crashing. Verified in local proof.
• CNAME emitted alongside index.html. Correct for GitHub Pages custom domain.
• site/dist/ added to .gitignore — build output not committed. Correct.

deploy-pages.yml:

• Push-to-main only, scoped to results/, site/, schema/**. No workflow_dispatch, schedule, or pull_request triggers. A submission PR can never trigger deploy. Correct isolation.
• Actions SHA-pinned (same vetted SHAs as validate.yml/schema-sync.yml for checkout/setup-node; upload-pages-artifact v5.0.0 and deploy-pages v5.0.0 pinned to full commit SHAs).
• Permissions: contents:read, pages:write, id-token:write. No secrets. OIDC auth for Pages deploy. Minimal and correct.
• concurrency: pages group, cancel-in-progress: false — never cancel a mid-publish deploy. Good.
• GITHUB_SHA passed via env (same pattern as validate.yml). No ${{ }} interpolation in run blocks.

README update: Correct — links bench.tps.dev, restates model-pure caveat, explains the deploy trigger. Repo layout tree updated.

Deploy timing note: Accurate — whichever of PR #2 or #3 lands second triggers the first real non-empty deploy. If #3 merges first, empty-state renders (no crash). Correct.

No concerns. Approving.

[tps-sherlock]

Security Review — APPROVE ✅

New deploy surface (GitHub Pages for bench.tps.dev). All five points verified:

1. Trigger: push-to-main only, paths-scoped

on:
  push:
    branches:
      - main
    paths:
      - "results/**"
      - "site/**"
      - "schema/**"

No workflow_dispatch, no schedule, no pull_request, no pull_request_target. A submission PR only ever exercises validate.yml — it never touches Pages. Deployment only happens as a direct consequence of something landing on main. ✅

2. Permissions: minimal, top-level

permissions:
  contents: read
  pages: write
  id-token: write

contents: read to build, pages: write + id-token: write to publish via OIDC. Nothing else. No actions: write, no deployments: write, no packages: write. ✅

3. All actions SHA-pinned to full commit SHAs

uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6.4.0
uses: actions/upload-pages-artifact@fc324d3547104276b827a68afc52ff2a11cc49c9 # v5.0.0
uses: actions/deploy-pages@cd2ce8fcbc39b97be8ca5fce6e763baed58fa128 # v5.0.0

Checkout and setup-node reuse this repo's existing vetted pins from validate.yml/schema-sync.yml. Upload and deploy are pinned to their v5.0.0 release SHAs. ✅

4. Zero secrets

No secrets.* references anywhere in the workflow. The Pages deploy action authenticates via the workflow's own OIDC token — no stored credential. ✅

5. Build script: local-only, no network, no exec

site/build.mjs:

• Reads: results/**/*.json and schema/ — repo-local JSON files only
• Writes: site/dist/ — local output directory
• No network calls: no fetch, no http, no child_process, no exec
• Schema-validates every result before use — same ajv strict validator as validate-results.mjs, same schema/versions.json discriminator
• escapeHtml() on all user-controlled strings before HTML output — model names, host labels, quant strings, timestamps all escaped
• Skip-and-warn on invalid files — a malformed/legacy result file never takes the site down
• No client-side JS — pure static HTML, native <details>/<summary> for per-kind breakdowns
• GITHUB_SHA via env: env: GITHUB_SHA: ${{ github.sha }} — passed as environment variable, not interpolated into run: body ✅

Additional observations

• Concurrency: group: "pages", cancel-in-progress: false — prevents race conditions on deploy, never cancels mid-publish ✅
• Environment: github-pages with URL output — standard GitHub Pages pattern ✅
• CNAME: bench.tps.dev — custom domain, standard ✅
• No XSS surface: all dynamic content goes through escapeHtml(), no innerHTML, no event handlers, no inline scripts ✅

Verdict

Clean. Well-designed deploy surface with defense in depth: trigger scoping, minimal permissions, SHA-pinned actions, zero secrets, and a build script that only reads local files and escapes all output.

[tps-sherlock]

Security Review: PR #3 — static matrix build + Pages deploy

Verdict: APPROVE. No security issues found. This is a well-constructed deploy surface.

Trigger isolation ✅

The workflow fires on push to main only, path-scoped to results/**, site/**, schema/**:

on:
  push:
    branches:
      - main
    paths:
      - "results/**"
      - "site/**"
      - "schema/**"

No workflow_dispatch, schedule, pull_request, or pull_request_target. A submission PR (which triggers validate.yml on pull_request) can never reach this workflow. The attack surface is exactly what lands on main — and only when the build inputs actually change.

Permissions ✅

Top-level, minimal:

permissions:
  contents: read
  pages: write
  id-token: write

No actions: write, no packages, no deployments beyond Pages. The id-token: write is required for OIDC-based Pages auth and is scoped to the job, not a persistent credential.

Action pins ✅

All four actions are SHA-pinned to full commit hashes:

uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6.4.0
uses: actions/upload-pages-artifact@fc324d3547104276b827a68afc52ff2a11cc49c9 # v5.0.0
uses: actions/deploy-pages@cd2ce8fcbc39b97be8ca5fce6e763baed58fa128 # v5.0.0

checkout and setup-node reuse the same pins from validate.yml/schema-sync.yml. upload-pages-artifact and deploy-pages are pinned to their v5.0.0 release SHAs.

Zero secrets ✅

No secrets.* reference anywhere in the workflow. Pages deploy authenticates via the workflow's own OIDC token — no stored credential to leak or rotate.

Build script: no network, no exec, no untrusted input ✅

site/build.mjs is purely local filesystem:

import { readFileSync, readdirSync, statSync, mkdirSync, writeFileSync } from "node:fs";

• Reads: results/**/*.json and schema/ (local JSON only)
• Writes: site/dist/index.html and site/dist/CNAME
• No fetch, no child_process, no exec, no import() of dynamic modules
• No network calls of any kind

XSS hardening ✅

All user-controlled data (model names, host labels, quant strings) passes through escapeHtml() before rendering:

function escapeHtml(s) {
  return String(s)
    .replace(/&/g, "&")
    .replace(/</g, "&lt;")
    .replace(/>/g, "&gt;")
    .replace(/"/g, "&quot;")
    .replace(/'/g, "&#39;");
}

The Kern banner is a hardcoded constant, not user-controlled. The CUSTOM_DOMAIN is hardcoded. The blobUrl() function constructs URLs from GITHUB_SHA (env, trusted) and relPath (derived from relative() on real filesystem paths — can't contain traversal).

Path validation ✅

The result file walker validates the directory structure before processing:

if (segs.length !== 4 || segs[0] !== "results") {
  console.warn(`[site/build] skip: "${relPath}" does not match results/<model-slug>/<host-label>/<file>.json`);
  continue;
}

Only files matching results/<slug>/<label>/<file>.json are processed. Anything outside that structure is skipped with a warning.

Graceful degradation ✅

Invalid JSON, unknown toolVersion, and schema validation failures all result in skip + warn, never a crash:

console.warn(`[site/build] skip: "${relPath}" is not valid JSON — ${e.message}`);
console.warn(`[site/build] skip: "${relPath}" toolVersion "${doc?.toolVersion}" has no entry in schema/versions.json`);
console.warn(`[site/build] skip: "${relPath}" failed schema validation — ${msg}`);

A malformed result file can't take the site down.

Concurrency guard ✅

concurrency:
  group: "pages"
  cancel-in-progress: false

Prevents mid-publish cancellation — a deploy in flight completes before the next one starts.

Summary

This is a textbook example of a well-scoped GitHub Pages deploy workflow. The trigger is narrow, permissions are minimal, actions are SHA-pinned, no secrets are used, the build script has no network surface, and all user-controlled data is HTML-escaped. No changes requested.