Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion oc-cmd-compare/AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -142,6 +142,7 @@ Everything below was a real bug. Keep them in mind when touching `src/data/scrap
`normalizeKey` lowercases, drops vendor prefix and parentheticals, strips punctuation, then applies `ALIASES` table in `constants/data.ts`. Add alias whenever model appears on one side only from branding (check `mpc --check`, lists `only in OpenCode` / `only in CommandCode`). OpenCode's free `… Free` rows drop the trailing marker in `normalizeKey` itself (the suffix is systematic, docs-only); the CommandCode free-handling is a separate mechanism (rate-only pass + `fillDeals`), not a key alias. Speed variants (GLM-5.2 Fast, Kimi K2.7 Code HighSpeed, MiMo V2.6 Pro UltraSpeed) intentionally distinct keys.

- **AA slugs carry what the catalogs drop, so those need aliases too** — a vendor prefix (`reflection-beam`, `tencent-hy4`) and a bake date (`mimo-v2-5-0424`) both miss the bare key, leaving a scored model with no ability; `Jev 1.14.0` is the same CommandCode build listed twice and aliases back to `jev` so the row collapses.
- **AA also writes the bake date and effort variant into its slug** (`deepseek-v4-1-flash-1017-high`, scored 32.6/28.6/18.5 across max/high/non-reasoning) — without stripping both, DeepSeek read as unscored while AA had three scores for it; `aaKeyVariants` strips them only when a 3-4 digit date is present, because without that anchor the effort rule would eat a real name token (`Qwen3.8-Max`).
- **OpenCode `Beam Free` is CommandCode `Beam 1.0`** — the trailing `Free` is stripped generically, so the bare `beam` key aliases to `beam10`; without it the same model showed twice, free on each side of a tie.

## Metrics semantics
Expand Down Expand Up @@ -173,7 +174,7 @@ Everything below was a real bug. Keep them in mind when touching `src/data/scrap
- **Fill order after lead: same benchmark's keyless `aa-web` scrape, then CommandCode (last resort).** `aa` primary merges `aa-web` (index + speed) then `cc`; `cc` primary merges AA (API when keyed, else `aa-web`) — CC's pages dropped `Tok/s` (Sep 2026), so this is where a `cc` primary gets its speed; `aa-web`/`file`/`url` fall back to `cc`. `--no-fallback` disables fills entirely.
- `data/sources/aa/web.ts` is keyless and **live**: the models page embeds its chart top-N in the Next.js RSC flight payload (`self.__next_f.push([1,"…"])`), decoded and bracket-matched at the `"initialModels"` array. Each record carries `slug`/`name`/`intelligenceIndex` and a median output speed under `timescaleData.medianOutputSpeed`, so the scrape supplies **both** index and tps. Only that embedded top-N is covered, so `aa-web` is **partial**; full coverage needs `AA_API_KEY` (`--bench aa`), **paginated** (`pagination.has_more`, page_size 200) — follow every page or most models silently miss. API row shape `{name, slug, evaluations.artificial_analysis_intelligence_index, performance.median_output_tokens_per_second}`, one row per reasoning variant, so keep best per key. AA slugs normalize cleanly via `normalizeKey` (`glm-5-3-flash` -> `glm53flash`); slug keys win over label keys. `parseAaWeb` stays a pure parser over input text (fixture-pinned, no network in tests).
- **`aa` reads the free list at `/api/v2/language/models/free`, never `/api/v2/data/llms/models`** — AA retired that subtree on 2026-11-04 (410 Gone) and the paid list answers 403 without Pro, while the free rows carry the same `evaluations.artificial_analysis_intelligence_index` and `performance.median_output_tokens_per_second`, paginated by `pagination.has_more`.
- **No cache anywhere in mpc — the benchmark layer is fetched live every run.** (Was: `$XDG_CACHE_HOME/mpc/ability-aa-web.json`, 7d TTL, plus a whole-result `mpc/ability.json` cache and the `--refresh` flag that existed only to bust them. A run that parsed 0 scores persisted the empty set, so every run inside the TTL inherited it — a transient page problem became silent, week-long data loss. Files, TTL const, flag all removed.)
- **No TTL cache anywhere in mpc — the benchmark layer is fetched live every run.** (Was: `$XDG_CACHE_HOME/mpc/ability-aa-web.json`, 7d TTL, plus a whole-result `mpc/ability.json` cache and the `--refresh` flag that existed only to bust them. A run that parsed 0 scores persisted the empty set, so every run inside the TTL inherited it — a transient page problem became silent, week-long data loss. Files, TTL const, flag all removed.) **One exception: `data/sources/aa/lastGood.ts` is a failure fallback, not a cache** — written only after a successful fetch with scores, read only when the live call throws (the free tier is 100 requests/day and used to blank a third of the table mid-day), and labelled `last good, Nh old — API: …` so a stale replay is never mistaken for a fresh one. An empty answer is never written.
- **AA scores are versioned in-band and the field name never changes** — `/language/models/free` carries `intelligence_index_version` (v5.0.1 since 2026-10-27), so a v4/v5 mix reads as models moving; the footer label prints it, and a 429/401/410 falls back to the keyless page instead of failing the run. A published figure beats a derived one: `TPS_OVERRIDES` pins Ultrafast at 1000 rather than 2x its base.
- CC's `Intelligence` and AA's index are _same_ benchmark (agree to ~2dp). CC publishes no `Tok/s`, so a `cc` primary always fills speed from AA; a keyed `aa` primary that already has both returns without a `cc` round-trip.
- Ability and speed ride one `BenchData` (`{intelligence, tps}`): `cc` covers ability only, `aa-web`/`aa` cover both. A model with no speed figure gets neutral 0.5 in VAL, not a penalty.
Expand Down
2 changes: 1 addition & 1 deletion oc-cmd-compare/package.json
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"name": "@jeffreyjyz/mpc",
"version": "0.1.10",
"version": "0.1.11",
"description": "Compare model pricing across opencode Go and Command Code plans",
"license": "MIT",
"type": "module",
Expand Down
18 changes: 16 additions & 2 deletions oc-cmd-compare/src/data/bench/resolve.ts
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
import { fetchText } from "#~/data/scrape/index.ts";
import { loadAaApi } from "#~/data/sources/aa/api.ts";
import { hydrate, readLastGood } from "#~/data/sources/aa/lastGood.ts";
import { loadAaWeb } from "#~/data/sources/aa/web.ts";
import { normalizeKey } from "#~/keys.ts";
import { loadCc } from "./cc.ts";
Expand Down Expand Up @@ -75,9 +76,22 @@ export async function resolvePrimary(opts: AbilityOptions): Promise<Resolved> {
})`,
};
} catch (error) {
// A spent daily quota, a rotated key or a retired path must not break every
// run: the keyless page carries the same index, just for fewer models.
// A spent daily quota, a rotated key or a retired path must not blank a third
// of the table: replay the last complete answer, and say how old it is. Only
// when there is no stored answer does the keyless page scrape take over.
const reason = error instanceof Error ? error.message : String(error);
const cached = await readLastGood();
if (cached && cached.intelligence.length > 0) {
const ageHours = Math.max(
0,
Math.round((Date.now() - cached.fetchedAt) / 3_600_000),
);
return {
scheme: "aa",
data: hydrate(cached),
label: `Artificial Analysis (last good, ${ageHours}h old — API: ${reason})`,
};
}
const data = await loadAaWeb();
if (data.intelligence.size === 0) throw error;
return {
Expand Down
7 changes: 6 additions & 1 deletion oc-cmd-compare/src/data/sources/aa/api.ts
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
import { API_URL } from "#~/constants/sources.ts";
import { parseAaApi } from "./parse.ts";
import { writeLastGood } from "./lastGood.ts";

/** Full catalog via the Artificial Analysis API. Needs a key. Paginated. */
export async function loadAaApi(
Expand All @@ -21,7 +22,11 @@ export async function loadAaApi(
if (!pagination?.has_more) break;
}

return indexVersion ? { intelligence, tps, indexVersion } : { intelligence, tps };
const result = indexVersion ? { intelligence, tps, indexVersion } : { intelligence, tps };
// Keep the last complete answer: the free tier runs out mid-day and the
// fallback scrape covers far fewer models.
await writeLastGood(result);
return result;
}

async function fetchTextWithKey(url: string, key: string): Promise<string> {
Expand Down
76 changes: 76 additions & 0 deletions oc-cmd-compare/src/data/sources/aa/lastGood.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,76 @@
// Last-good Artificial Analysis payload.
//
// The free tier allows 100 requests per 24 h, so a busy day exhausts it and every
// later run used to fall back to the partial page scrape — a third of the table
// losing its ability and speed mid-day. This keeps the last complete answer on
// disk and replays it ONLY when the live call fails; a fresh answer always wins,
// and the label says how old the replay is.
//
// It is a failure fallback, not a TTL cache: nothing is served from it while the
// API answers, and an empty result is never written (the bug that killed the old
// cache: one bad run persisted an empty set and every later run inherited it).
import { mkdir, readFile, writeFile } from "node:fs/promises";
import { homedir } from "node:os";
import { join } from "node:path";

export type LastGood = {
fetchedAt: number;
indexVersion?: string;
intelligence: Array<[string, number]>;
tps: Array<[string, number]>;
};

function path(): string {
const base = process.env.XDG_CACHE_HOME ?? join(homedir(), ".cache");
return join(base, "mpc", "aa-last-good.json");
}

export async function readLastGood(): Promise<LastGood | undefined> {
try {
const parsed = JSON.parse(await readFile(path(), "utf8")) as LastGood;
if (!Array.isArray(parsed.intelligence) || parsed.intelligence.length === 0)
return undefined;
return parsed;
} catch {
return undefined;
}
}

export async function writeLastGood(data: {
indexVersion?: string;
intelligence: Map<string, number>;
tps: Map<string, number>;
}): Promise<void> {
if (data.intelligence.size === 0) return; // never persist an empty answer
try {
const target = path();
await mkdir(join(target, ".."), { recursive: true });
const body: LastGood = {
fetchedAt: Date.now(),
indexVersion: data.indexVersion,
intelligence: [...data.intelligence.entries()],
tps: [...data.tps.entries()],
};
await writeFile(target, JSON.stringify(body));
} catch {
// best effort: a read-only cache dir must not fail a run
}
}

/** Rehydrate a stored payload into the loader's shape. */
export function hydrate(cached: LastGood): {
intelligence: Map<string, number>;
tps: Map<string, number>;
indexVersion?: string;
} {
return {
intelligence: new Map(cached.intelligence),
tps: new Map(cached.tps),
indexVersion: cached.indexVersion,
};
}

/** Cache path, for diagnostics and tests. */
export function lastGoodPath(): string {
return path();
}
8 changes: 6 additions & 2 deletions oc-cmd-compare/src/data/sources/aa/parse.ts
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
import { normalizeKey } from "#~/keys.ts";
import { aaKeyVariants } from "#~/keys.ts";

/** Pure parser over one AA API page, for tests and the loader. */
export function parseAaApi(body: unknown): {
Expand Down Expand Up @@ -26,7 +26,11 @@ export function parseAaApi(body: unknown): {
const record = row as Record<string, unknown>;
const name = String(record.name ?? record.slug ?? record.id ?? "");
const slug = typeof record.slug === "string" ? record.slug : undefined;
const keys = [slug ? normalizeKey(slug) : "", normalizeKey(name)].filter(Boolean);
// AA writes the bake date and the effort variant into its own slugs, so a row
// answers to several keys; every one is registered and the best score wins.
const keys = [
...new Set([...(slug ? aaKeyVariants(slug) : []), ...aaKeyVariants(name)]),
].filter(Boolean);
if (keys.length === 0) continue;

const iq = findNumber(record, /intelligence_index/i);
Expand Down
27 changes: 27 additions & 0 deletions oc-cmd-compare/src/keys.ts
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,33 @@ export function normalizeKey(name: string): string {
return ALIASES[s] ?? s;
}

/**
* Keys an Artificial Analysis row may answer to. AA writes the bake date and the
* effort variant into the slug itself (`deepseek-v4-1-flash-1017-high`) while the
* catalogs name the model bare (`DeepSeek V4.1 Flash`), so the exact key never
* matches and a scored model reads as unscored.
*
* The strippings only run when a 3-4 digit bake date is present: without that
* anchor they would eat a real name token (`Qwen3.8-Max` -> `qwen38`).
*/
export function aaKeyVariants(name: string): string[] {
const out = [normalizeKey(name)];
const dated = /[\s_-]+\d{3,4}(?:$|[\s_-])/i.test(name);
if (!dated) return out;
const noDate = name.replace(/[\s_-]+\d{3,4}(?=$|[\s_-])/i, "");
for (const candidate of [
noDate,
noDate.replace(
/[\s_-]+(?:non[\s_-]*reasoning|reasoning|minimal|medium|xhigh|high|low|max)$/i,
"",
),
]) {
const key = normalizeKey(candidate);
if (!out.includes(key)) out.push(key);
}
return out;
}

/** Display name: strip parentheticals and collapse whitespace. */
export function displayName(name: string): string {
return name
Expand Down
17 changes: 16 additions & 1 deletion oc-cmd-compare/test/unit/keys.test.ts
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
import { describe, expect, test } from "bun:test";
import { BOUNDARY } from "#~/constants/data.ts";
import { displayName, normalizeKey } from "#~/keys.ts";
import { aaKeyVariants, displayName, normalizeKey } from "#~/keys.ts";

describe("normalizeKey", () => {
test("strips vendor prefix and punctuation", () => {
Expand Down Expand Up @@ -60,3 +60,18 @@ describe("displayName", () => {
expect(displayName(`Kimi K3${BOUNDARY}`)).toBe("Kimi K3");
});
});

describe("aaKeyVariants", () => {
test("strips AA bake dates and effort suffixes", () => {
expect(aaKeyVariants("deepseek-v4-1-flash-1017-high")).toContain("deepseekv41flash");
expect(aaKeyVariants("deepseek-v4-1-flash-1017-high")).toContain("deepseekv41flashhigh");
expect(aaKeyVariants("deepseek-v4-pro-0813")).toContain("deepseekv4pro");
// The exact key stays first, so a variant row still answers to its own slug.
expect(aaKeyVariants("deepseek-v4-1-flash-1017")[0]).toBe("deepseekv41flash1017");
});
test("leaves undated slugs untouched", () => {
// No bake date: stripping "max" here would map Qwen 3.8 Max onto Qwen 3.8.
expect(aaKeyVariants("qwen3-8-max")).toEqual(["qwen38max"]);
expect(aaKeyVariants("claude-sonnet-5-5-medium")).toEqual(["claudesonnet55medium"]);
});
});
29 changes: 29 additions & 0 deletions oc-cmd-compare/test/unit/lastGood.test.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
import { describe, expect, test } from "bun:test";
import { mkdtempSync, existsSync } from "node:fs";
import { tmpdir } from "node:os";
import { join } from "node:path";
import { hydrate, lastGoodPath, readLastGood, writeLastGood } from "#~/data/sources/aa/lastGood.ts";

describe("AA last-good cache", () => {
test("round-trips a scored payload", async () => {
process.env.XDG_CACHE_HOME = mkdtempSync(join(tmpdir(), "mpc-lg-"));
await writeLastGood({
indexVersion: "v5.0.1",
intelligence: new Map([["deepseekv41flash", 32.6]]),
tps: new Map([["deepseekv41flash", 251]]),
});
const cached = await readLastGood();
expect(cached?.indexVersion).toBe("v5.0.1");
if (!cached) throw new Error("no cached payload");
const data = hydrate(cached);
expect(data.intelligence.get("deepseekv41flash")).toBe(32.6);
expect(data.tps.get("deepseekv41flash")).toBe(251);
});

test("never persists an empty answer", async () => {
process.env.XDG_CACHE_HOME = mkdtempSync(join(tmpdir(), "mpc-lg-"));
await writeLastGood({ intelligence: new Map(), tps: new Map() });
expect(existsSync(lastGoodPath())).toBe(false);
expect(await readLastGood()).toBeUndefined();
});
});
Loading