diff --git a/oc-cmd-compare/AGENTS.md b/oc-cmd-compare/AGENTS.md index 2c2dae9..c08c483 100644 --- a/oc-cmd-compare/AGENTS.md +++ b/oc-cmd-compare/AGENTS.md @@ -142,6 +142,7 @@ Everything below was a real bug. Keep them in mind when touching `src/data/scrap `normalizeKey` lowercases, drops vendor prefix and parentheticals, strips punctuation, then applies `ALIASES` table in `constants/data.ts`. Add alias whenever model appears on one side only from branding (check `mpc --check`, lists `only in OpenCode` / `only in CommandCode`). OpenCode's free `… Free` rows drop the trailing marker in `normalizeKey` itself (the suffix is systematic, docs-only); the CommandCode free-handling is a separate mechanism (rate-only pass + `fillDeals`), not a key alias. Speed variants (GLM-5.2 Fast, Kimi K2.7 Code HighSpeed, MiMo V2.6 Pro UltraSpeed) intentionally distinct keys. - **AA slugs carry what the catalogs drop, so those need aliases too** — a vendor prefix (`reflection-beam`, `tencent-hy4`) and a bake date (`mimo-v2-5-0424`) both miss the bare key, leaving a scored model with no ability; `Jev 1.14.0` is the same CommandCode build listed twice and aliases back to `jev` so the row collapses. +- **AA also writes the bake date and effort variant into its slug** (`deepseek-v4-1-flash-1017-high`, scored 32.6/28.6/18.5 across max/high/non-reasoning) — without stripping both, DeepSeek read as unscored while AA had three scores for it; `aaKeyVariants` strips them only when a 3-4 digit date is present, because without that anchor the effort rule would eat a real name token (`Qwen3.8-Max`). - **OpenCode `Beam Free` is CommandCode `Beam 1.0`** — the trailing `Free` is stripped generically, so the bare `beam` key aliases to `beam10`; without it the same model showed twice, free on each side of a tie. ## Metrics semantics @@ -173,7 +174,7 @@ Everything below was a real bug. Keep them in mind when touching `src/data/scrap - **Fill order after lead: same benchmark's keyless `aa-web` scrape, then CommandCode (last resort).** `aa` primary merges `aa-web` (index + speed) then `cc`; `cc` primary merges AA (API when keyed, else `aa-web`) — CC's pages dropped `Tok/s` (Sep 2026), so this is where a `cc` primary gets its speed; `aa-web`/`file`/`url` fall back to `cc`. `--no-fallback` disables fills entirely. - `data/sources/aa/web.ts` is keyless and **live**: the models page embeds its chart top-N in the Next.js RSC flight payload (`self.__next_f.push([1,"…"])`), decoded and bracket-matched at the `"initialModels"` array. Each record carries `slug`/`name`/`intelligenceIndex` and a median output speed under `timescaleData.medianOutputSpeed`, so the scrape supplies **both** index and tps. Only that embedded top-N is covered, so `aa-web` is **partial**; full coverage needs `AA_API_KEY` (`--bench aa`), **paginated** (`pagination.has_more`, page_size 200) — follow every page or most models silently miss. API row shape `{name, slug, evaluations.artificial_analysis_intelligence_index, performance.median_output_tokens_per_second}`, one row per reasoning variant, so keep best per key. AA slugs normalize cleanly via `normalizeKey` (`glm-5-3-flash` -> `glm53flash`); slug keys win over label keys. `parseAaWeb` stays a pure parser over input text (fixture-pinned, no network in tests). - **`aa` reads the free list at `/api/v2/language/models/free`, never `/api/v2/data/llms/models`** — AA retired that subtree on 2026-11-04 (410 Gone) and the paid list answers 403 without Pro, while the free rows carry the same `evaluations.artificial_analysis_intelligence_index` and `performance.median_output_tokens_per_second`, paginated by `pagination.has_more`. -- **No cache anywhere in mpc — the benchmark layer is fetched live every run.** (Was: `$XDG_CACHE_HOME/mpc/ability-aa-web.json`, 7d TTL, plus a whole-result `mpc/ability.json` cache and the `--refresh` flag that existed only to bust them. A run that parsed 0 scores persisted the empty set, so every run inside the TTL inherited it — a transient page problem became silent, week-long data loss. Files, TTL const, flag all removed.) +- **No TTL cache anywhere in mpc — the benchmark layer is fetched live every run.** (Was: `$XDG_CACHE_HOME/mpc/ability-aa-web.json`, 7d TTL, plus a whole-result `mpc/ability.json` cache and the `--refresh` flag that existed only to bust them. A run that parsed 0 scores persisted the empty set, so every run inside the TTL inherited it — a transient page problem became silent, week-long data loss. Files, TTL const, flag all removed.) **One exception: `data/sources/aa/lastGood.ts` is a failure fallback, not a cache** — written only after a successful fetch with scores, read only when the live call throws (the free tier is 100 requests/day and used to blank a third of the table mid-day), and labelled `last good, Nh old — API: …` so a stale replay is never mistaken for a fresh one. An empty answer is never written. - **AA scores are versioned in-band and the field name never changes** — `/language/models/free` carries `intelligence_index_version` (v5.0.1 since 2026-10-27), so a v4/v5 mix reads as models moving; the footer label prints it, and a 429/401/410 falls back to the keyless page instead of failing the run. A published figure beats a derived one: `TPS_OVERRIDES` pins Ultrafast at 1000 rather than 2x its base. - CC's `Intelligence` and AA's index are _same_ benchmark (agree to ~2dp). CC publishes no `Tok/s`, so a `cc` primary always fills speed from AA; a keyed `aa` primary that already has both returns without a `cc` round-trip. - Ability and speed ride one `BenchData` (`{intelligence, tps}`): `cc` covers ability only, `aa-web`/`aa` cover both. A model with no speed figure gets neutral 0.5 in VAL, not a penalty. diff --git a/oc-cmd-compare/package.json b/oc-cmd-compare/package.json index d8c176a..04363ae 100644 --- a/oc-cmd-compare/package.json +++ b/oc-cmd-compare/package.json @@ -1,6 +1,6 @@ { "name": "@jeffreyjyz/mpc", - "version": "0.1.10", + "version": "0.1.11", "description": "Compare model pricing across opencode Go and Command Code plans", "license": "MIT", "type": "module", diff --git a/oc-cmd-compare/src/data/bench/resolve.ts b/oc-cmd-compare/src/data/bench/resolve.ts index 122a0cb..a902d90 100644 --- a/oc-cmd-compare/src/data/bench/resolve.ts +++ b/oc-cmd-compare/src/data/bench/resolve.ts @@ -1,5 +1,6 @@ import { fetchText } from "#~/data/scrape/index.ts"; import { loadAaApi } from "#~/data/sources/aa/api.ts"; +import { hydrate, readLastGood } from "#~/data/sources/aa/lastGood.ts"; import { loadAaWeb } from "#~/data/sources/aa/web.ts"; import { normalizeKey } from "#~/keys.ts"; import { loadCc } from "./cc.ts"; @@ -75,9 +76,22 @@ export async function resolvePrimary(opts: AbilityOptions): Promise { })`, }; } catch (error) { - // A spent daily quota, a rotated key or a retired path must not break every - // run: the keyless page carries the same index, just for fewer models. + // A spent daily quota, a rotated key or a retired path must not blank a third + // of the table: replay the last complete answer, and say how old it is. Only + // when there is no stored answer does the keyless page scrape take over. const reason = error instanceof Error ? error.message : String(error); + const cached = await readLastGood(); + if (cached && cached.intelligence.length > 0) { + const ageHours = Math.max( + 0, + Math.round((Date.now() - cached.fetchedAt) / 3_600_000), + ); + return { + scheme: "aa", + data: hydrate(cached), + label: `Artificial Analysis (last good, ${ageHours}h old — API: ${reason})`, + }; + } const data = await loadAaWeb(); if (data.intelligence.size === 0) throw error; return { diff --git a/oc-cmd-compare/src/data/sources/aa/api.ts b/oc-cmd-compare/src/data/sources/aa/api.ts index 177a145..a7a9a90 100644 --- a/oc-cmd-compare/src/data/sources/aa/api.ts +++ b/oc-cmd-compare/src/data/sources/aa/api.ts @@ -1,5 +1,6 @@ import { API_URL } from "#~/constants/sources.ts"; import { parseAaApi } from "./parse.ts"; +import { writeLastGood } from "./lastGood.ts"; /** Full catalog via the Artificial Analysis API. Needs a key. Paginated. */ export async function loadAaApi( @@ -21,7 +22,11 @@ export async function loadAaApi( if (!pagination?.has_more) break; } - return indexVersion ? { intelligence, tps, indexVersion } : { intelligence, tps }; + const result = indexVersion ? { intelligence, tps, indexVersion } : { intelligence, tps }; + // Keep the last complete answer: the free tier runs out mid-day and the + // fallback scrape covers far fewer models. + await writeLastGood(result); + return result; } async function fetchTextWithKey(url: string, key: string): Promise { diff --git a/oc-cmd-compare/src/data/sources/aa/lastGood.ts b/oc-cmd-compare/src/data/sources/aa/lastGood.ts new file mode 100644 index 0000000..6b91bc2 --- /dev/null +++ b/oc-cmd-compare/src/data/sources/aa/lastGood.ts @@ -0,0 +1,76 @@ +// Last-good Artificial Analysis payload. +// +// The free tier allows 100 requests per 24 h, so a busy day exhausts it and every +// later run used to fall back to the partial page scrape — a third of the table +// losing its ability and speed mid-day. This keeps the last complete answer on +// disk and replays it ONLY when the live call fails; a fresh answer always wins, +// and the label says how old the replay is. +// +// It is a failure fallback, not a TTL cache: nothing is served from it while the +// API answers, and an empty result is never written (the bug that killed the old +// cache: one bad run persisted an empty set and every later run inherited it). +import { mkdir, readFile, writeFile } from "node:fs/promises"; +import { homedir } from "node:os"; +import { join } from "node:path"; + +export type LastGood = { + fetchedAt: number; + indexVersion?: string; + intelligence: Array<[string, number]>; + tps: Array<[string, number]>; +}; + +function path(): string { + const base = process.env.XDG_CACHE_HOME ?? join(homedir(), ".cache"); + return join(base, "mpc", "aa-last-good.json"); +} + +export async function readLastGood(): Promise { + try { + const parsed = JSON.parse(await readFile(path(), "utf8")) as LastGood; + if (!Array.isArray(parsed.intelligence) || parsed.intelligence.length === 0) + return undefined; + return parsed; + } catch { + return undefined; + } +} + +export async function writeLastGood(data: { + indexVersion?: string; + intelligence: Map; + tps: Map; +}): Promise { + if (data.intelligence.size === 0) return; // never persist an empty answer + try { + const target = path(); + await mkdir(join(target, ".."), { recursive: true }); + const body: LastGood = { + fetchedAt: Date.now(), + indexVersion: data.indexVersion, + intelligence: [...data.intelligence.entries()], + tps: [...data.tps.entries()], + }; + await writeFile(target, JSON.stringify(body)); + } catch { + // best effort: a read-only cache dir must not fail a run + } +} + +/** Rehydrate a stored payload into the loader's shape. */ +export function hydrate(cached: LastGood): { + intelligence: Map; + tps: Map; + indexVersion?: string; +} { + return { + intelligence: new Map(cached.intelligence), + tps: new Map(cached.tps), + indexVersion: cached.indexVersion, + }; +} + +/** Cache path, for diagnostics and tests. */ +export function lastGoodPath(): string { + return path(); +} diff --git a/oc-cmd-compare/src/data/sources/aa/parse.ts b/oc-cmd-compare/src/data/sources/aa/parse.ts index a27604c..df1859b 100644 --- a/oc-cmd-compare/src/data/sources/aa/parse.ts +++ b/oc-cmd-compare/src/data/sources/aa/parse.ts @@ -1,4 +1,4 @@ -import { normalizeKey } from "#~/keys.ts"; +import { aaKeyVariants } from "#~/keys.ts"; /** Pure parser over one AA API page, for tests and the loader. */ export function parseAaApi(body: unknown): { @@ -26,7 +26,11 @@ export function parseAaApi(body: unknown): { const record = row as Record; const name = String(record.name ?? record.slug ?? record.id ?? ""); const slug = typeof record.slug === "string" ? record.slug : undefined; - const keys = [slug ? normalizeKey(slug) : "", normalizeKey(name)].filter(Boolean); + // AA writes the bake date and the effort variant into its own slugs, so a row + // answers to several keys; every one is registered and the best score wins. + const keys = [ + ...new Set([...(slug ? aaKeyVariants(slug) : []), ...aaKeyVariants(name)]), + ].filter(Boolean); if (keys.length === 0) continue; const iq = findNumber(record, /intelligence_index/i); diff --git a/oc-cmd-compare/src/keys.ts b/oc-cmd-compare/src/keys.ts index 06d9c6b..fdc620b 100644 --- a/oc-cmd-compare/src/keys.ts +++ b/oc-cmd-compare/src/keys.ts @@ -24,6 +24,33 @@ export function normalizeKey(name: string): string { return ALIASES[s] ?? s; } +/** + * Keys an Artificial Analysis row may answer to. AA writes the bake date and the + * effort variant into the slug itself (`deepseek-v4-1-flash-1017-high`) while the + * catalogs name the model bare (`DeepSeek V4.1 Flash`), so the exact key never + * matches and a scored model reads as unscored. + * + * The strippings only run when a 3-4 digit bake date is present: without that + * anchor they would eat a real name token (`Qwen3.8-Max` -> `qwen38`). + */ +export function aaKeyVariants(name: string): string[] { + const out = [normalizeKey(name)]; + const dated = /[\s_-]+\d{3,4}(?:$|[\s_-])/i.test(name); + if (!dated) return out; + const noDate = name.replace(/[\s_-]+\d{3,4}(?=$|[\s_-])/i, ""); + for (const candidate of [ + noDate, + noDate.replace( + /[\s_-]+(?:non[\s_-]*reasoning|reasoning|minimal|medium|xhigh|high|low|max)$/i, + "", + ), + ]) { + const key = normalizeKey(candidate); + if (!out.includes(key)) out.push(key); + } + return out; +} + /** Display name: strip parentheticals and collapse whitespace. */ export function displayName(name: string): string { return name diff --git a/oc-cmd-compare/test/unit/keys.test.ts b/oc-cmd-compare/test/unit/keys.test.ts index 085963b..e9f5204 100644 --- a/oc-cmd-compare/test/unit/keys.test.ts +++ b/oc-cmd-compare/test/unit/keys.test.ts @@ -1,6 +1,6 @@ import { describe, expect, test } from "bun:test"; import { BOUNDARY } from "#~/constants/data.ts"; -import { displayName, normalizeKey } from "#~/keys.ts"; +import { aaKeyVariants, displayName, normalizeKey } from "#~/keys.ts"; describe("normalizeKey", () => { test("strips vendor prefix and punctuation", () => { @@ -60,3 +60,18 @@ describe("displayName", () => { expect(displayName(`Kimi K3${BOUNDARY}`)).toBe("Kimi K3"); }); }); + +describe("aaKeyVariants", () => { + test("strips AA bake dates and effort suffixes", () => { + expect(aaKeyVariants("deepseek-v4-1-flash-1017-high")).toContain("deepseekv41flash"); + expect(aaKeyVariants("deepseek-v4-1-flash-1017-high")).toContain("deepseekv41flashhigh"); + expect(aaKeyVariants("deepseek-v4-pro-0813")).toContain("deepseekv4pro"); + // The exact key stays first, so a variant row still answers to its own slug. + expect(aaKeyVariants("deepseek-v4-1-flash-1017")[0]).toBe("deepseekv41flash1017"); + }); + test("leaves undated slugs untouched", () => { + // No bake date: stripping "max" here would map Qwen 3.8 Max onto Qwen 3.8. + expect(aaKeyVariants("qwen3-8-max")).toEqual(["qwen38max"]); + expect(aaKeyVariants("claude-sonnet-5-5-medium")).toEqual(["claudesonnet55medium"]); + }); +}); diff --git a/oc-cmd-compare/test/unit/lastGood.test.ts b/oc-cmd-compare/test/unit/lastGood.test.ts new file mode 100644 index 0000000..35ef281 --- /dev/null +++ b/oc-cmd-compare/test/unit/lastGood.test.ts @@ -0,0 +1,29 @@ +import { describe, expect, test } from "bun:test"; +import { mkdtempSync, existsSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { hydrate, lastGoodPath, readLastGood, writeLastGood } from "#~/data/sources/aa/lastGood.ts"; + +describe("AA last-good cache", () => { + test("round-trips a scored payload", async () => { + process.env.XDG_CACHE_HOME = mkdtempSync(join(tmpdir(), "mpc-lg-")); + await writeLastGood({ + indexVersion: "v5.0.1", + intelligence: new Map([["deepseekv41flash", 32.6]]), + tps: new Map([["deepseekv41flash", 251]]), + }); + const cached = await readLastGood(); + expect(cached?.indexVersion).toBe("v5.0.1"); + if (!cached) throw new Error("no cached payload"); + const data = hydrate(cached); + expect(data.intelligence.get("deepseekv41flash")).toBe(32.6); + expect(data.tps.get("deepseekv41flash")).toBe(251); + }); + + test("never persists an empty answer", async () => { + process.env.XDG_CACHE_HOME = mkdtempSync(join(tmpdir(), "mpc-lg-")); + await writeLastGood({ intelligence: new Map(), tps: new Map() }); + expect(existsSync(lastGoodPath())).toBe(false); + expect(await readLastGood()).toBeUndefined(); + }); +});