Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
68 changes: 68 additions & 0 deletions .github/workflows/deploy-pages.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,68 @@
name: Deploy Pages

# Push-to-main ONLY, and only when the build's actual inputs change. This
# workflow is the ONLY path that can publish to bench.tps.dev — a submission
# PR (validate.yml, pull_request-triggered) never runs this, never touches
# Pages. No workflow_dispatch, no schedule, no pull_request_target: a
# deployment only ever happens as a direct consequence of something landing
# on main.
on:
push:
branches:
- main
paths:
- "results/**"
- "site/**"
- "schema/**"

# Minimal, top-level: contents:read to build, pages:write + id-token:write to
# publish via OIDC. No secrets are used anywhere in this workflow — the
# GitHub Pages deploy action authenticates via the workflow's own OIDC token,
# not a stored credential.
permissions:
contents: read
pages: write
id-token: write

# Only one deployment in flight at a time; never cancel one mid-publish.
concurrency:
group: "pages"
cancel-in-progress: false

jobs:
build:
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0

- name: Set up Node
uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6.4.0
with:
node-version: "22"

- name: Install dependencies
run: npm ci

# Reads every results/**/*.json, schema-validates, emits site/dist/.
# Skip-and-warn on invalid result files — never fatal.
- name: Build site
env:
GITHUB_SHA: ${{ github.sha }}
run: node site/build.mjs

- name: Upload Pages artifact
uses: actions/upload-pages-artifact@fc324d3547104276b827a68afc52ff2a11cc49c9 # v5.0.0
with:
path: site/dist

deploy:
needs: build
runs-on: ubuntu-latest
environment:
name: github-pages
url: ${{ steps.deployment.outputs.page_url }}
steps:
- name: Deploy to GitHub Pages
id: deployment
uses: actions/deploy-pages@cd2ce8fcbc39b97be8ca5fce6e763baed58fa128 # v5.0.0
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
node_modules/
*.gguf
*.tgz
site/dist/
12 changes: 11 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -59,7 +59,15 @@ Each result reports precision@3 and MRR, aggregate and per query-kind (`stress`/

## Where this data is used

The accumulated `results/` matrix is the data source for the (upcoming) benchmark matrix site and an agent-queryable MCP surface — both read straight from this repo's `results/` directory; there's no separate database to keep in sync.
The accumulated `results/` matrix is the data source for [bench.tps.dev](https://bench.tps.dev) (below) and an (upcoming) agent-queryable MCP surface — both read straight from this repo's `results/` directory; there's no separate database to keep in sync.

## bench.tps.dev — the matrix site

[bench.tps.dev](https://bench.tps.dev) is a static, server-rendered page: every `results/**/*.json` in this repo, schema-validated and pivoted into a model × host matrix (precision@3, MRR, ms/embed, peak RSS, backend — with a per-row `<details>` for the per-kind breakdown). No client framework, no client-side JS, no database — the page is regenerated from scratch on every relevant push.

It carries the same model-pure caveat as above, as a banner on the page itself, not a footnote: these numbers are exact-cosine recall for the embedding model alone, not a prediction of Flair's production (HNSW + BM25-hybrid) recall.

**How it updates**: `.github/workflows/deploy-pages.yml` runs `site/build.mjs` and redeploys to GitHub Pages on every push to `main` that touches `results/**`, `site/**`, or `schema/**` — so a merged submission PR (or a schema/site change) is live within a couple of minutes, with no manual step. It never runs against a PR branch; a submission PR only ever exercises `validate.yml`. Invalid or unparseable result files are skipped with a warning rather than failing the build.

## Repo layout

Expand All @@ -73,6 +81,8 @@ scripts/
lib/
forbidden-content.mjs hostname/path/username content sweep
naming.mjs the results/<model-slug>/<host-label>/<utc-iso>.json convention
site/
build.mjs static build for bench.tps.dev — reads results/, emits site/dist/
results/
<model-slug>/<host-label>/<utc-iso>.json submitted results
```
Loading
Loading