Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -127,7 +127,7 @@ Tests must be hermetic: CI has no `cmduse` binary and no homebrew prefix, so any
- **Verify with the scripts, never by eye:** `opencode-plugin/scripts/verify-crate.ts <version>` (crate ⇄ release assets ⇄ tap pins) and `verify-release.ts <version>` for npm (`--source` for a CI-published package, `--dir <path>` for another package). Running them from the repo root fails — the scripts live in `opencode-plugin/scripts/`.
- **A plugin tarball that hashes differently from the local pack is usually bun's bundle comments, not a bad publish** — `bun build` writes `// <module path>` per inlined module, and the path differs between a workspace install and CI's npm one; the build now strips them (`scripts/strip-bundle-comments.ts`), so diff the bundles before treating it as a fault (2026-11-06).
- **Publish every npm package by tag, not from a terminal** — one tag names one package: `plugin-v*` opencode-plugin, `shell-rc-v*` opencode-shell-rc, `context-v*` opencode-context, `session-dir-v*` opencode-session-dir, `mpc-v*` oc-cmd-compare, `reqshape-v*` reqshape. Bump that package's `package.json` (PATCH for incremental work, MINOR for a line-level change; each is versioned independently of the Rust crates), commit, `git tag <prefix>X.Y.Z && git push origin <tag>`. `npm-plugin.yml` resolves the tag to a directory, runs that package's test/typecheck/build, refuses a tag that disagrees with `package.json`, publishes through npm OIDC trusted publishing (no 2FA prompt, no long-lived token), then polls until the tarball is served. **Each package needs its own trusted-publisher entry on npmjs.com** (owner, repo, bare workflow filename `npm-plugin.yml`) — one workflow for all six is what keeps that filename identical everywhere. The manifest's `repository` must name this repo or npm rejects provenance; **stage/approve (`npm stage publish` → `npm stage approve`) is the fallback**, never bare `npm publish`: a `202` there can leave a ghost version that packument and `npm stage list` both deny. Rebuild after every bump — `dist/` embeds the version string.
- **Wait for the tag's own run.** `gh run list --limit 1` returns whatever started last; select by `headBranch == <tag>` before bumping the tap, or a chained script trusts the previous tag's success.
- **Wait for the tag's own run.** `gh run list --limit 1` returns whatever started last; select by `headBranch == <tag>` before bumping the tap, or a chained script trusts the previous tag's success. **Same for any consumer pin:** `dist-tags` answering the new version is not the verdict — flipping the plugin pin while the run was `in_progress` left an empty 64 B `~/.cache/opencode/npm/<pkg>@<ver>/` the loader treats as installed, so delete that dir after the publish lands rather than pinning early (2026-11-08).

## Known traps

Expand Down
1 change: 1 addition & 0 deletions oc-cmd-compare/AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -168,6 +168,7 @@ Everything below was a real bug. Keep them in mind when touching `src/data/scrap
- `data/bench/` (`index.ts`/`resolve.ts`) resolves benchmark scores; `--bench` picks scheme. `constants/cli.ts`'s `BENCH_DEFAULT` makes default **contextual: `aa` when AA key present (`AA_API_KEY` or `--aa-key`), else `cc`**. Explicit `--bench` always wins — a key only chooses default, upgrades API path.
- **Fill order after lead: same benchmark's keyless `aa-web` scrape, then CommandCode (last resort).** `aa` primary merges `aa-web` (index + speed) then `cc`; `cc` primary merges AA (API when keyed, else `aa-web`) — CC's pages dropped `Tok/s` (Sep 2026), so this is where a `cc` primary gets its speed; `aa-web`/`file`/`url` fall back to `cc`. `--no-fallback` disables fills entirely.
- `data/sources/aa/web.ts` is keyless and **live**: the models page embeds its chart top-N in the Next.js RSC flight payload (`self.__next_f.push([1,"…"])`), decoded and bracket-matched at the `"initialModels"` array. Each record carries `slug`/`name`/`intelligenceIndex` and a median output speed under `timescaleData.medianOutputSpeed`, so the scrape supplies **both** index and tps. Only that embedded top-N is covered, so `aa-web` is **partial**; full coverage needs `AA_API_KEY` (`--bench aa`), **paginated** (`pagination.has_more`, page_size 200) — follow every page or most models silently miss. API row shape `{name, slug, evaluations.artificial_analysis_intelligence_index, performance.median_output_tokens_per_second}`, one row per reasoning variant, so keep best per key. AA slugs normalize cleanly via `normalizeKey` (`glm-5-3-flash` -> `glm53flash`); slug keys win over label keys. `parseAaWeb` stays a pure parser over input text (fixture-pinned, no network in tests).
- **`aa` reads the free list at `/api/v2/language/models/free`, never `/api/v2/data/llms/models`** — AA retired that subtree on 2026-11-04 (410 Gone) and the paid list answers 403 without Pro, while the free rows carry the same `evaluations.artificial_analysis_intelligence_index` and `performance.median_output_tokens_per_second`, paginated by `pagination.has_more`.
- **No cache anywhere in mpc — the benchmark layer is fetched live every run.** (Was: `$XDG_CACHE_HOME/mpc/ability-aa-web.json`, 7d TTL, plus a whole-result `mpc/ability.json` cache and the `--refresh` flag that existed only to bust them. A run that parsed 0 scores persisted the empty set, so every run inside the TTL inherited it — a transient page problem became silent, week-long data loss. Files, TTL const, flag all removed.)
- CC's `Intelligence` and AA's index are _same_ benchmark (agree to ~2dp). CC publishes no `Tok/s`, so a `cc` primary always fills speed from AA; a keyed `aa` primary that already has both returns without a `cc` round-trip.
- Ability and speed ride one `BenchData` (`{intelligence, tps}`): `cc` covers ability only, `aa-web`/`aa` cover both. A model with no speed figure gets neutral 0.5 in VAL, not a penalty.
Expand Down
2 changes: 1 addition & 1 deletion oc-cmd-compare/package.json
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"name": "@jeffreyjyz/mpc",
"version": "0.1.6",
"version": "0.1.7",
"description": "Compare model pricing across opencode Go and Command Code plans",
"license": "MIT",
"type": "module",
Expand Down
5 changes: 4 additions & 1 deletion oc-cmd-compare/src/constants/sources.ts
Original file line number Diff line number Diff line change
Expand Up @@ -79,7 +79,10 @@ export const DEFAULT_OC_PLAN = "go";
export const DOC_URL = "https://opencode.ai/v2/docs/console/go/";
export const OC_MODELS_URL = "https://opencode.ai/zen/go/v1/models";

export const API_URL = "https://artificialanalysis.ai/api/v2/data/llms/models";
/** Artificial Analysis retired `/api/v2/data/*` on 2026-11-04 (410 Gone). The free
* tier serves the same index and median output speed mpc reads, so no Pro plan
* is needed; the paid list at `/api/v2/language/models` answers 403 without one. */
export const API_URL = "https://artificialanalysis.ai/api/v2/language/models/free";
export const AA_MODELS_URL = "https://artificialanalysis.ai/models";

// CommandCode only publishes Intelligence and Tok/s on the GOAT/Pro catalogs.
Expand Down
16 changes: 13 additions & 3 deletions oc-cmd-compare/src/data/sources/aa/api.ts
Original file line number Diff line number Diff line change
Expand Up @@ -26,13 +26,23 @@ async function fetchTextWithKey(url: string, key: string): Promise<string> {
headers: { "x-api-key": key, "user-agent": "mpc/0.1" },
});
if (!res.ok) {
throw new Error(
`Artificial Analysis API ${res.status}: set AA_API_KEY or use --bench cc / aa-web`,
);
throw new Error(`Artificial Analysis API ${res.status}: ${statusHint(res.status)}`);
}
return res.text();
}

/** One line per cause: a retired path is not a bad key, and the old message
* blamed credentials for every status. */
function statusHint(status: number): string {
if (status === 401) return "AA_API_KEY rejected, replace the key";
if (status === 403)
return "this endpoint needs a Pro subscription, the free list is /language/models/free";
if (status === 404 || status === 410)
return "endpoint retired, see artificialanalysis.ai/data-api";
if (status === 429) return "daily quota spent, use --bench cc / aa-web";
return "unexpected response, use --bench cc / aa-web";
}

/** Keep the best variant per key; AA lists a row per reasoning effort. */
function mergeAaPage(
page: { intelligence: Map<string, number>; tps: Map<string, number> },
Expand Down
Loading