Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions oc-cmd-compare/AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -113,6 +113,7 @@ Same prune hits every `bun link` in group (`reqshape` too); global symlink store
- Contracts to preserve:
- `buildMetrics` invariant: `payPerRequest * requestsPerMonth === plan.price`.
- Free models: `costPerRequest === 0` ⇒ `requestsPerMonth = Infinity`, `index = 100`. JSON has no `Infinity`, so `renderJson` (plain `JSON.stringify` of `rows`) emits `requestsPerMonth: null` — and `requestsPerFiveHour`/`requestsPerWeek` with it. **`null` means unbounded, not unknown.** Consumer treating it as missing data (or comparing numerically) wrong; `sortProjections` in reqshape guards it explicitly for this reason.
- **A free model wins VAL volume term, never VAL itself** — infinite requests is the best volume, but the row still enters the composite, so a weak free model no longer reads 100 (Beam 1.0 at ability 17.7 went 100 → 82, behind three DeepSeek rows).
- One shared index scale across both providers (`buildRows` scores concatenated list).
- Keep README user-facing; agent/design notes here.
- **Size limits (pragmatic, not a game):**
Expand Down Expand Up @@ -140,6 +141,8 @@ Everything below was a real bug. Keep them in mind when touching `src/data/scrap

`normalizeKey` lowercases, drops vendor prefix and parentheticals, strips punctuation, then applies `ALIASES` table in `constants/data.ts`. Add alias whenever model appears on one side only from branding (check `mpc --check`, lists `only in OpenCode` / `only in CommandCode`). OpenCode's free `… Free` rows drop the trailing marker in `normalizeKey` itself (the suffix is systematic, docs-only); the CommandCode free-handling is a separate mechanism (rate-only pass + `fillDeals`), not a key alias. Speed variants (GLM-5.2 Fast, Kimi K2.7 Code HighSpeed, MiMo V2.6 Pro UltraSpeed) intentionally distinct keys.

- **AA slugs carry what the catalogs drop, so those need aliases too** — a vendor prefix (`reflection-beam`, `tencent-hy4`) and a bake date (`mimo-v2-5-0424`) both miss the bare key, leaving a scored model with no ability; `Jev 1.14.0` is the same CommandCode build listed twice and aliases back to `jev` so the row collapses.

## Metrics semantics

- `payPerRequest = planPrice * costPerRequest / allowance` — plan-relative. Cheaper subscription can post lower per-request cost while buying fewer requests, so `$/1K` effective rate, `req/mo` volume signal.
Expand Down
2 changes: 1 addition & 1 deletion oc-cmd-compare/package.json
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"name": "@jeffreyjyz/mpc",
"version": "0.1.8",
"version": "0.1.9",
"description": "Compare model pricing across opencode Go and Command Code plans",
"license": "MIT",
"type": "module",
Expand Down
14 changes: 7 additions & 7 deletions oc-cmd-compare/src/cli/engine/score.ts
Original file line number Diff line number Diff line change
Expand Up @@ -76,19 +76,19 @@ export function assignValueIndex(metrics: EntryMetrics[], config: ScoreConfig):
const byAbility = of(scored, ability);
const bySpeed = of(speeded, speed);

// Unpriced models have unlimited requests, so they sit at the top of the
// scale and stay out of the range below.
// Unlimited requests win the volume term outright — you cannot buy more than
// infinite — but the model still enters the composite, so a free model with a
// low ability no longer outranks everything by virtue of being free.
const volumeTerm = (m: EntryMetrics) =>
Number.isFinite(m.requestsPerMonth) ? (byVolume.get(m) ?? 0.5) : 1;

const composites = new Map<EntryMetrics, number>();
for (const m of scored) {
if (!Number.isFinite(m.requestsPerMonth)) {
m.valueIndex = 100;
continue;
}
composites.set(
m,
100 * (wAbility ?? 0) * (byAbility.get(m) ?? 0.5) +
100 * (wTps ?? 0) * (bySpeed.get(m) ?? 0.5) +
100 * (wVolume ?? 0) * (byVolume.get(m) ?? 0.5) +
100 * (wVolume ?? 0) * volumeTerm(m) +
100 * (wCache ?? 0) * (1 - (byCache.get(m) ?? 0.5)) +
100 * (wOutput ?? 0) * (1 - (byOutput.get(m) ?? 0.5)),
);
Expand Down
7 changes: 7 additions & 0 deletions oc-cmd-compare/src/constants/data.ts
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,13 @@ export const ALIASES: Record<string, string> = {
// CommandCode prefixes these with "Tencent"; OpenCode Go does not.
tencenthy3: "hy3",
tencenthy4preview: "hy4preview",
// AA's slug keeps the vendor the catalogs drop: "tencent-hy4", "reflection-beam".
tencenthy4: "hy4",
reflectionbeam: "beam10",
// AA suffixes the bake date onto the slug: "mimo-v2-5-0424".
mimov250424: "mimov25",
// CommandCode lists the same Jev build twice, once with its version suffix.
jev1140: "jev",
// CommandCode writes "(exp)"; OpenCode Go writes "-exp".
deepseekv4flashvision: "deepseekv4flashvisionexp",
// Alibaba's "Qwen3.8-Flash-Next" is the release name of the model
Expand Down
5 changes: 4 additions & 1 deletion oc-cmd-compare/src/constants/view.ts
Original file line number Diff line number Diff line change
Expand Up @@ -100,7 +100,10 @@ export const LEGEND: [string, string][] = [
["ability", "benchmark score for the model (source above)"],
["tps", "output tokens per second (source above)"],
["COST", "0-100 volume index: requests/mo, lower is better (no ability)"],
["VAL", "0-100 ability-aware value: ability + speed + volume + cache + output"],
[
"VAL",
"0-100 ability-aware value: ability + speed + volume + cache + output · a free row wins volume only, so it can post COST 0 and a mid VAL",
],
];

export const CSV_HEADER = [
Expand Down
28 changes: 28 additions & 0 deletions oc-cmd-compare/test/unit/core/score.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -102,4 +102,32 @@ describe("ability value index", () => {
}
}
});

test("a free model wins volume, not the whole index", () => {
// Free means unlimited requests, which is the best possible volume; it must not
// also erase ability, tps and rates, or a weak free model outranks everything.
const paid = { input: 1, output: 4, cacheRead: 0.1, cacheWrite: null };
const free = { input: 0, output: 0, cacheRead: 0, cacheWrite: null };
const entries = [
entry({ key: "strong", pricing: paid }),
entry({ key: "weakfree", pricing: free }),
];
const ability = new Map([
["strong", 72],
["weakfree", 12],
]);
const metrics = buildMetrics(
entries,
new Map([["cc", ccPlan]]),
workloads,
ability,
new Map(),
DEFAULT_SCORE,
);
const byKey = new Map(entries.map((e, i) => [e.key, metrics[i]]));
const weakFree = byKey.get("weakfree");
expect(weakFree?.requestsPerMonth).toBe(Number.POSITIVE_INFINITY);
expect(weakFree?.valueIndex).toBeLessThan(100);
expect(byKey.get("strong")?.valueIndex).toBeGreaterThan(weakFree?.valueIndex ?? 0);
});
});
7 changes: 7 additions & 0 deletions oc-cmd-compare/test/unit/keys.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,13 @@ describe("normalizeKey", () => {
test("aliases vendor-branded variants", () => {
expect(normalizeKey("Tencent Hy3")).toBe("hy3");
expect(normalizeKey("Tencent Hy4 Preview")).toBe("hy4preview");
// AA keeps the vendor in its slug; the catalogs do not.
expect(normalizeKey("tencent-hy4")).toBe(normalizeKey("Hy4"));
expect(normalizeKey("reflection-beam")).toBe(normalizeKey("Beam 1.0"));
// AA appends the bake date to the slug.
expect(normalizeKey("mimo-v2-5-0424")).toBe(normalizeKey("MiMo-V2.5"));
// CommandCode lists the same Jev build with and without its version.
expect(normalizeKey("Jev 1.14.0")).toBe(normalizeKey("Jev"));
expect(normalizeKey("DeepSeek V4 Flash Vision (exp)")).toBe("deepseekv4flashvisionexp");
// Alibaba's release name for the model CommandCode sells as "Qwen 3.8
// Flash", so the AA index lands on the row that is listed here.
Expand Down
Loading