Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions oc-cmd-compare/AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -142,6 +142,7 @@ Everything below was a real bug. Keep them in mind when touching `src/data/scrap
`normalizeKey` lowercases, drops vendor prefix and parentheticals, strips punctuation, then applies `ALIASES` table in `constants/data.ts`. Add alias whenever model appears on one side only from branding (check `mpc --check`, lists `only in OpenCode` / `only in CommandCode`). OpenCode's free `… Free` rows drop the trailing marker in `normalizeKey` itself (the suffix is systematic, docs-only); the CommandCode free-handling is a separate mechanism (rate-only pass + `fillDeals`), not a key alias. Speed variants (GLM-5.2 Fast, Kimi K2.7 Code HighSpeed, MiMo V2.6 Pro UltraSpeed) intentionally distinct keys.

- **AA slugs carry what the catalogs drop, so those need aliases too** — a vendor prefix (`reflection-beam`, `tencent-hy4`) and a bake date (`mimo-v2-5-0424`) both miss the bare key, leaving a scored model with no ability; `Jev 1.14.0` is the same CommandCode build listed twice and aliases back to `jev` so the row collapses.
- **OpenCode `Beam Free` is CommandCode `Beam 1.0`** — the trailing `Free` is stripped generically, so the bare `beam` key aliases to `beam10`; without it the same model showed twice, free on each side of a tie.

## Metrics semantics

Expand Down Expand Up @@ -173,6 +174,7 @@ Everything below was a real bug. Keep them in mind when touching `src/data/scrap
- `data/sources/aa/web.ts` is keyless and **live**: the models page embeds its chart top-N in the Next.js RSC flight payload (`self.__next_f.push([1,"…"])`), decoded and bracket-matched at the `"initialModels"` array. Each record carries `slug`/`name`/`intelligenceIndex` and a median output speed under `timescaleData.medianOutputSpeed`, so the scrape supplies **both** index and tps. Only that embedded top-N is covered, so `aa-web` is **partial**; full coverage needs `AA_API_KEY` (`--bench aa`), **paginated** (`pagination.has_more`, page_size 200) — follow every page or most models silently miss. API row shape `{name, slug, evaluations.artificial_analysis_intelligence_index, performance.median_output_tokens_per_second}`, one row per reasoning variant, so keep best per key. AA slugs normalize cleanly via `normalizeKey` (`glm-5-3-flash` -> `glm53flash`); slug keys win over label keys. `parseAaWeb` stays a pure parser over input text (fixture-pinned, no network in tests).
- **`aa` reads the free list at `/api/v2/language/models/free`, never `/api/v2/data/llms/models`** — AA retired that subtree on 2026-11-04 (410 Gone) and the paid list answers 403 without Pro, while the free rows carry the same `evaluations.artificial_analysis_intelligence_index` and `performance.median_output_tokens_per_second`, paginated by `pagination.has_more`.
- **No cache anywhere in mpc — the benchmark layer is fetched live every run.** (Was: `$XDG_CACHE_HOME/mpc/ability-aa-web.json`, 7d TTL, plus a whole-result `mpc/ability.json` cache and the `--refresh` flag that existed only to bust them. A run that parsed 0 scores persisted the empty set, so every run inside the TTL inherited it — a transient page problem became silent, week-long data loss. Files, TTL const, flag all removed.)
- **AA scores are versioned in-band and the field name never changes** — `/language/models/free` carries `intelligence_index_version` (v5.0.1 since 2026-10-27), so a v4/v5 mix reads as models moving; the footer label prints it, and a 429/401/410 falls back to the keyless page instead of failing the run. A published figure beats a derived one: `TPS_OVERRIDES` pins Ultrafast at 1000 rather than 2x its base.
- CC's `Intelligence` and AA's index are _same_ benchmark (agree to ~2dp). CC publishes no `Tok/s`, so a `cc` primary always fills speed from AA; a keyed `aa` primary that already has both returns without a `cc` round-trip.
- Ability and speed ride one `BenchData` (`{intelligence, tps}`): `cc` covers ability only, `aa-web`/`aa` cover both. A model with no speed figure gets neutral 0.5 in VAL, not a penalty.
- `VAL` uses `abilityWeight` (default 0.35) and `tpsWeight` (default 0.10); remaining weight splits volume/cache/output 50/25/25. Unscored models get `valueIndex = null`; never coerce missing score to zero.
Expand Down
2 changes: 1 addition & 1 deletion oc-cmd-compare/package.json
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"name": "@jeffreyjyz/mpc",
"version": "0.1.9",
"version": "0.1.10",
"description": "Compare model pricing across opencode Go and Command Code plans",
"license": "MIT",
"type": "module",
Expand Down
5 changes: 4 additions & 1 deletion oc-cmd-compare/src/cli/engine/ability.ts
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
import { SPEED_SUFFIXES, SPEED_TPS_FACTOR } from "#~/constants/scoring.ts";
import { SPEED_SUFFIXES, SPEED_TPS_FACTOR, TPS_OVERRIDES } from "#~/constants/scoring.ts";

/**
* The base key a speed variant shares weights with, or undefined. FlashX is the
Expand Down Expand Up @@ -41,6 +41,9 @@ export function lookupTps(
suffixes: string[] = SPEED_SUFFIXES,
factor: number = SPEED_TPS_FACTOR,
): number | null {
// A published figure wins over anything derived, including the borrow below.
const stated = TPS_OVERRIDES[key];
if (stated !== undefined) return stated;
const direct = scores.get(key);
if (direct !== undefined) return direct;
const base = baseKey(key, suffixes);
Expand Down
3 changes: 3 additions & 0 deletions oc-cmd-compare/src/constants/data.ts
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,9 @@ export const ALIASES: Record<string, string> = {
// AA's slug keeps the vendor the catalogs drop: "tencent-hy4", "reflection-beam".
tencenthy4: "hy4",
reflectionbeam: "beam10",
// OpenCode's "Beam Free" is the same model CommandCode sells as "Beam 1.0";
// the trailing "Free" is stripped generically, so the bare key is what lands here.
beam: "beam10",
// AA suffixes the bake date onto the slug: "mimo-v2-5-0424".
mimov250424: "mimov25",
// CommandCode lists the same Jev build twice, once with its version suffix.
Expand Down
12 changes: 10 additions & 2 deletions oc-cmd-compare/src/constants/scoring.ts
Original file line number Diff line number Diff line change
Expand Up @@ -3,8 +3,9 @@ import type { ScoreConfig } from "#~/cli/engine/score.ts";
/** Tokens per million, the denominator of every per-M-token rate. */
export const PER_MILLION = 1_000_000;

/** Speed-variant suffixes that share the base model's weights. */
export const SPEED_SUFFIXES = ["ultraspeed", "highspeed", "fastx", "fast"];
/** Speed-variant suffixes that share the base model's weights. Longest first:
* `ultrafast` must be tried before `fast`, or the base key keeps `ultra`. */
export const SPEED_SUFFIXES = ["ultraspeed", "highspeed", "ultrafast", "fastx", "fast"];

/**
* Throughput a speed variant without its own figure borrows from its base.
Expand All @@ -15,6 +16,13 @@ export const SPEED_SUFFIXES = ["ultraspeed", "highspeed", "fastx", "fast"];
*/
export const SPEED_TPS_FACTOR = 2;

/** Published throughput for a model that neither source scores. The suffix rule
* would guess 512 for Ultrafast (2x its base); CommandCode advertises 1000, and a
* stated figure always beats a derived one. */
export const TPS_OVERRIDES: Record<string, number> = {
deepseekv41flashultrafast: 1000,
};

export const DEFAULT_SCORE: ScoreConfig = {
abilityWeight: 0.35,
tpsWeight: 0.1,
Expand Down
35 changes: 25 additions & 10 deletions oc-cmd-compare/src/data/bench/resolve.ts
Original file line number Diff line number Diff line change
Expand Up @@ -60,17 +60,32 @@ export async function resolvePrimary(opts: AbilityOptions): Promise<Resolved> {
if (!key) {
throw new Error("--bench aa needs a key: set AA_API_KEY or pass --aa-key");
}
const data = await loadAaApi(key);
if (data.intelligence.size === 0) {
throw new Error(
"Artificial Analysis returned no scores — check the API key and response shape",
);
try {
const data = await loadAaApi(key);
if (data.intelligence.size === 0) {
throw new Error(
"Artificial Analysis returned no scores — check the key and response shape",
);
}
return {
scheme: "aa",
data,
label: `Artificial Analysis (${data.intelligence.size} models, ${data.tps.size} speed${
data.indexVersion ? `, index ${data.indexVersion}` : ""
})`,
};
} catch (error) {
// A spent daily quota, a rotated key or a retired path must not break every
// run: the keyless page carries the same index, just for fewer models.
const reason = error instanceof Error ? error.message : String(error);
const data = await loadAaWeb();
if (data.intelligence.size === 0) throw error;
return {
scheme: "aa-web",
data,
label: `Artificial Analysis (web, partial — API: ${reason})`,
};
}
return {
scheme: "aa",
data,
label: `Artificial Analysis (${data.intelligence.size} models, ${data.tps.size} speed)`,
};
}
case "file":
if (!arg) throw new Error("--bench file:<path> needs a path");
Expand Down
3 changes: 3 additions & 0 deletions oc-cmd-compare/src/data/bench/types.ts
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,9 @@ export interface BenchData {
intelligence: Map<string, number>;
/** Output tokens per second, when the source publishes it. */
tps: Map<string, number>;
/** AA's index version for these scores (`v5.0.1`); absent for other sources.
* Scores are only comparable within one major version. */
indexVersion?: string;
}

export interface AbilityResult extends BenchData {
Expand Down
9 changes: 6 additions & 3 deletions oc-cmd-compare/src/data/sources/aa/api.ts
Original file line number Diff line number Diff line change
Expand Up @@ -4,21 +4,24 @@ import { parseAaApi } from "./parse.ts";
/** Full catalog via the Artificial Analysis API. Needs a key. Paginated. */
export async function loadAaApi(
key: string,
): Promise<{ intelligence: Map<string, number>; tps: Map<string, number> }> {
): Promise<{ intelligence: Map<string, number>; tps: Map<string, number>; indexVersion?: string }> {
const intelligence = new Map<string, number>();
const tps = new Map<string, number>();
const pageSize = 200;
const maxPages = 20;
let indexVersion: string | undefined;

for (let page = 1; page <= maxPages; page++) {
const res = await fetchTextWithKey(`${API_URL}?page=${page}&page_size=${pageSize}`, key);
const body = JSON.parse(res) as unknown;
mergeAaPage(parseAaApi(body), intelligence, tps);
const parsed = parseAaApi(body);
indexVersion ??= parsed.indexVersion;
mergeAaPage(parsed, intelligence, tps);
const pagination = (body as { pagination?: { has_more?: boolean } }).pagination;
if (!pagination?.has_more) break;
}

return { intelligence, tps };
return indexVersion ? { intelligence, tps, indexVersion } : { intelligence, tps };
}

async function fetchTextWithKey(url: string, key: string): Promise<string> {
Expand Down
9 changes: 8 additions & 1 deletion oc-cmd-compare/src/data/sources/aa/parse.ts
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ import { normalizeKey } from "#~/keys.ts";
export function parseAaApi(body: unknown): {
intelligence: Map<string, number>;
tps: Map<string, number>;
indexVersion?: string;
} {
const root = (body ?? {}) as Record<string, unknown>;
const rows = Array.isArray(root.data)
Expand All @@ -13,6 +14,12 @@ export function parseAaApi(body: unknown): {
: Object.values(root).filter((v) => v && typeof v === "object");
const intelligence = new Map<string, number>();
const tps = new Map<string, number>();
// AA versions the index in-band and keeps the field name stable, so the score is
// only interpretable beside this: v4 and v5 numbers are not comparable.
const indexVersion =
typeof root.intelligence_index_version === "string"
? root.intelligence_index_version
: undefined;

for (const row of rows) {
if (!row || typeof row !== "object") continue;
Expand All @@ -38,7 +45,7 @@ export function parseAaApi(body: unknown): {
}
}
}
return { intelligence, tps };
return indexVersion ? { intelligence, tps, indexVersion } : { intelligence, tps };
}

/** Depth-first search for the first finite number under a matching key. */
Expand Down
7 changes: 7 additions & 0 deletions oc-cmd-compare/test/data/bench.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -58,4 +58,11 @@ describe("parseAaApi", () => {
expect(tps.get("glm53flash")).toBeCloseTo(114.9, 2);
expect(intelligence.has("noscoresh")).toBe(false);
});

test("carries the index version, which is what makes scores comparable", () => {
const versioned = { ...BODY, intelligence_index_version: "v5.0.1" };
expect(parseAaApi(versioned).indexVersion).toBe("v5.0.1");
// Older payloads and fixtures carry no version; absence is not an error.
expect(parseAaApi(BODY).indexVersion).toBeUndefined();
});
});
2 changes: 2 additions & 0 deletions oc-cmd-compare/test/unit/keys.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,8 @@ describe("normalizeKey", () => {
expect(normalizeKey("mimo-v2-5-0424")).toBe(normalizeKey("MiMo-V2.5"));
// CommandCode lists the same Jev build with and without its version.
expect(normalizeKey("Jev 1.14.0")).toBe(normalizeKey("Jev"));
// OpenCode sells "Beam Free"; CommandCode the same model as "Beam 1.0".
expect(normalizeKey("Beam Free")).toBe(normalizeKey("Beam 1.0"));
expect(normalizeKey("DeepSeek V4 Flash Vision (exp)")).toBe("deepseekv4flashvisionexp");
// Alibaba's release name for the model CommandCode sells as "Qwen 3.8
// Flash", so the AA index lands on the row that is listed here.
Expand Down
Loading