Skip to content

fix(mpc): match AA's dated slugs, keep a last-good answer for quota days - #26

Merged
JeffreyJYZ merged 1 commit into
mainfrom
fix/aa-variant-keys
Nov 17, 2026
Merged

JeffreyJYZ merged 1 commit into
mainfrom
fix/aa-variant-keys

Conversation

@jyzbot

@jyzbot jyzbot commented Nov 16, 2026

Copy link
Copy Markdown
Collaborator

You were right — AA does score DeepSeek, and mpc was throwing the scores away.

The mismatch

AA writes the bake date and the effort variant into its slugs; the catalogs name the model bare:

deepseek-v4-1-flash-1017              → 32.6   (Max)
deepseek-v4-1-flash-1017-high         → 28.6
deepseek-v4-1-flash-1017-non-reasoning → 18.5
deepseek-v4-pro-0813 / -high / -non-reasoning → 23.1 / 20.8 / 12.6

normalizeKey("deepseek-v4-1-flash-1017-high") → deepseekv41flash1017high, which never equals deepseekv41flash, so 45 rows read as unscored while AA had them.

aaKeyVariants() now strips the bake date and then the effort suffix — only when a 3–4 digit date is present, because without that anchor the effort rule would eat a real name token (Qwen3.8-Max → qwen38). The exact key stays first.

Quota days

The free tier is 100 requests/24 h. A spent quota used to drop the whole table onto the partial page scrape, so ability and speed vanished mid-day:

ability  Artificial Analysis (web, partial — API: ... 429: daily quota spent)

lastGood.ts stores the last complete answer and replays it only when the live call throws, labelled last good, Nh old — API: …. An empty answer is never written — that is exactly what killed the old 7-day TTL cache.

Gates

bun test          119 pass, 0 fail  (+2 aaKeyVariants, +2 lastGood)
bun check         No type errors in 91 files
bun fmt --check   All 97 files are formatted
biome lint        1 info, no warnings

Version 0.1.11. Live proof of the ranking shift needs the quota to reset (429 right now), at which point the first successful run also populates the last-good file.

AA scores DeepSeek under dated variant slugs -- deepseek-v4-1-flash-1017 (32.6),
-1017-high (28.6), -1017-non-reasoning (18.5) -- while the catalogs name the model
bare, so normalizeKey never matched and DeepSeek read as unscored. aaKeyVariants
strips the bake date and the effort suffix, and only when a 3-4 digit date is
present: without that anchor the effort rule would eat a real name token
(Qwen3.8-Max -> qwen38).

The free tier is 100 requests/day, and a spent quota used to drop the table to the
partial page scrape, blanking a third of ability and speed mid-day. lastGood.ts
now stores the last complete answer and replays it only when the live call throws,
labelled with its age; an empty answer is never written, which is what killed the
old TTL cache.

Version 0.1.11.
@vercel

vercel Bot commented Nov 16, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
cmdcode-tools-site Ready Ready Preview Nov 16, 2026 10:33am UTC

@JeffreyJYZ
JeffreyJYZ merged commit e5c2535 into main Nov 17, 2026
10 checks passed
@JeffreyJYZ
JeffreyJYZ deleted the fix/aa-variant-keys branch November 17, 2026 02:02

This branch was successfully deployed

1 active deployment
Preview — edc1c7ee Deployed Nov 16, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants