OpenRouter Reliable Research Search

Does the way researchers call OpenRouter (and similar multi-provider routers) silently corrupt their results?

OpenRouter is a convenient unified API in front of dozens of inference providers. But by default it load-balances the same "model" across providers that may serve it quantized, truncated, differently-tokenized, or with silently-dropped sampling parameters. Two identical requests can hit two different backends and return materially different outputs. Most researchers who depend on OpenRouter for model outputs don't know this — and it can make a model look smarter or dumber, or shift its measured propensities, in ways that quietly invalidate a result.

This side-project:

1. Best practices — a thorough, citation-backed guide to using OpenRouter (or similar) reliably for research. → reports/openrouter-best-practices.md
2. A survey of important public research code (top ML conferences + LessWrong) that uses OpenRouter, checking whether their results could be silently corrupted, with a taxonomy of mistakes and a dataset. → findings/
3. An interactive explorer + a shareable summary image of the findings.
4. A Claude skill for using OpenRouter properly and auditing a codebase's usage. → skill/

Why this matters (the one-paragraph version)

By default OpenRouter routes each request to whichever provider is cheapest-and-up, weighted by the inverse square of price. Providers are free to serve a model at any quantization (down to int4), on any inference engine, and to silently ignore request parameters they don't support (temperature, seed, logprobs, response_format) unless you set require_parameters: true. Independent work finds inference-backend choice alone can move a benchmark score by up to ~16 percentage points, yet the inference stack is almost never reported. If your research pins none of this, "the model" you evaluated is a moving target.

This isn't a new claim — and that's the point

The mechanism is established. What was missing is evidence about whether working research code does anything about it. Four sources, each quoted verbatim and pinned in reports/prior-work.md:

• It is measurable, and it was measured at a top venue. Model Equality Testing (ICLR 2025) ran a two-sample test against commercial APIs and found 11 out of 31 endpoints serve different distributions than reference weights released by Meta.
• The gap can be enormous. On AIME25, identical gpt-oss-120b weights scored 93.3% via Cerebras/Nebius/Fireworks/DeepInfra/Novita/Together, 86.7% via Groq, 80.0% via Azure and 36.7% via CompactifAI — one slug, one benchmark, one week.
• The field does not notice. Chasing Shadows audited all 72 LLM-security papers at leading venues from 2023–2024. Its pitfall P9 — being unable to tell which model instance produced a result — was the most prevalent one it found, present in 73.6% (53) of papers, and not one of them discussed it.
• It has already invalidated published work. After a re-run changed only the OpenRouter provider, the author of the critiqued paper conceded his results "were contaminated by bad inference setups."

This survey is the missing piece: not "can routing corrupt a result" — that is settled — but how many real research repos leave the channel open. The answer is most of them.

Layout

Path	What's there
reports/	Best-practices guide; prior work; detection fingerprints; provider transparency audit; the provider A/B rerun experiment plan
findings/	The survey dataset (CSV/JSON), taxonomy of mistakes, fingerprint catalogue, methodology, claims.json
scripts/	Everything generated is generated here — see Provenance
artifact/	Interactive data explorer (self-contained HTML)
image/	Shareable summary graphic
skill/	use-openrouter-safely Claude skill
tests/	Enforces the provenance chain — prose numbers, artifact integrity, API field mappings

Headline findings

We audited 35 influential public repos (importance-first: NeurIPS · ICML · ICLR · ACL · NAACL · Nature + UK AISI · METR · Redwood · Palisade · Anthropic Fellows + LessWrong/AF) flagged as routing model calls through OpenRouter. Each verdict came from one audit agent plus one adversarial verifier reading the actual source. Numbers below are generated from findings/stats.json.

• 31 / 32 (97%) of the repos whose OpenRouter output actually reaches a reported number, a training set, or a safety measurement leave at least one uncontrolled provider-routing corruption channel open.

• Three denominators, stated plainly. 35 repos surveyed · 34 contain an OpenRouter call site anywhere · 32 put its output on a result path. The headline uses 32, because a repo only demonstrates something about using OpenRouter well if OpenRouter reaches a published number. Against the wider 34 the rate is 31/34 (91%).

• 113 specific claims/figures across 31 repos were traced to an OpenRouter-routed call and could be affected — each named down to the figure/table/number, with its mechanism and a "does this really depend on OpenRouter?" confidence. (34 high-impact, 53 medium, 26 low.)

• 23 carry a high-severity gap (can distort a result, not just reproducibility).

• Exactly one repo — nostalgebraist/cot_legibility — both uses OpenRouter for real results and controls for it (pins only:[novita], allow_fallbacks:False, logs the served provider). It's exemplary precisely because provider choice is its research question.

• The other three previously-"safe" repos are not success stories, and we no longer count them as such. Every row now carries a safety_class:

  class	n	in the headline denominator?	meaning
  at_risk	31	✅	OpenRouter feeds a reported result, with an uncontrolled channel open
  handled	1	✅	feeds a reported result and controls for it — the only real positive
  not_on_result_path	2	❌	OpenRouter present in the repo, but no reported result depends on it
  no_usage_found	1	❌	no OpenRouter call site at all (discovery false positive)

  A repo that never routes a research call through OpenRouter has demonstrated nothing about using OpenRouter well — lumping those in with the one genuine success overstated the good news. The two at_risk + handled classes are exactly the 32 repos flagged critical_route; a build check fails if that ever stops holding.

• Most pervasive gaps (all silent by default): no provenance logging (28), data-policy left open (27), unpinned quantization (26), probabilistic routing (26).

Read this correctly. "At risk" = the code leaves a known corruption channel open and uncontrolled, not that any published number is wrong. The possibly-impacted findings are hypotheses worth checking, not demonstrated errors. We audited how each repo routes model calls; we did not re-run experiments across providers to measure the actual delta. See findings/methodology.md.

All 35 rows completed adversarial verification — a second agent re-read the paper and code and was told to drop any cited figure it could not find in the real source. That stage earns its keep: it caught a first-pass agent inventing numbers for one repo, and a paper whose reproducibility appendix claims a model was called "via Google AI" when the code hardcodes an OpenRouter slug.

Can you tell from the outside?

The audit above reads code. A separate question is whether a reader — with only the paper, its figures, its appendix and whatever data was released — could ever notice. There is one famous case where someone did: nostalgebraist read the chain-of-thought transcripts published with an "illegible reasoning" paper, recognised them as decode-level gibberish rather than reasoning, re-ran the same prompts on a different OpenRouter provider, and watched the phenomenon vanish. The paper's author agreed his results "were contaminated by bad inference setups."

So we went looking for the rest of that method. 18 parallel research sweeps, one adversarial verifier per candidate, merged into 17 fingerprint families → reports/detection-fingerprints.md, data in findings/fingerprints.json. 13 rest on a documented catch rather than on reasoning; 12 are checkable only if the authors released raw outputs. The catalogue includes two explicit negative families — run-to-run variance, temperature-0 non-determinism, cost and throughput side channels — because each of them looks like a fingerprint and is not one.

Then we ran every one of the 35 surveyed projects against the catalogue, one agent per repo, with a second adversarial pass over every positive claim. The verdict lives in the dataset as Signs provider issues messed up the paper:

Verdict	Repos	Meaning
nothing_checkable_released	14	No transcripts, per-sample data, or provider metadata. No reader could ever check.
fingerprints_found	8	Something in their own published output is visibly off.
checked_clean	7	Enough was released to look, and the applicable checks come back clean.
inconclusive	6	Raw outputs exist but do not settle it either way.

Only 21 of 35 projects released anything a reader could check at all. Among those that did, the hit rate is not small — released outputs contain literal <|endoftext|> salad, completion walls below the declared max_tokens, near-half-empty completion rates on one model but not its neighbours, and <think> tags that appear for three hours and then stop. But the honest headline runs the other way: the fingerprint that mattered most is the one nobody publishes. Provider metadata survives in released data for only a handful of these projects, aggregate-only reporting destroys every distributional signal before publication, and the original catch itself needed both released transcripts and someone willing to pay to re-run the experiment.

Deliverables

📊 Interactive explorer	Provider Routing Inspector — filter by verdict/severity, click a mistake to filter, "Routing Roulette" demo, per-repo link to the exact offending line: https://claude.ai/code/artifact/6ba2006d-e72e-47b8-9e81-fe6270f8305e (source: artifact/index.html)
🖼️ Shareable image	image/openrouter_findings.png — one-glance summary + the fix
📄 Best-practices guide	reports/openrouter-best-practices.md
📚 Prior work	reports/prior-work.md — who already documented this, with every quotation pinned to its source
🔍 Detection fingerprints	reports/detection-fingerprints.md — 17 things a reader can look for in a published paper, and the much longer list of what cannot be told at all
🕵️ Provider transparency	reports/provider-transparency.md — could a researcher find out what changed by reading the vendor's own docs? Mostly not: 31% of sampled endpoints declare no quantization at all
🗂️ Dataset	findings/survey.csv · findings/survey.json (incl. a validated GitHub permalink to each call site)
🧾 Taxonomy	findings/taxonomy.md — the M1–M12 mistake catalog
🛠️ Claude skill	skill/use-openrouter-safely/ — guidance + a heuristic static auditor

Reproduce

# audit any repo for OpenRouter reliability mistakes (heuristic first pass)
uv run skill/use-openrouter-safely/scripts/audit_openrouter.py <path-to-repo>

Use the skill

use-openrouter-safely works in two modes: writing OpenRouter calls correctly, and auditing a repo's existing usage against the M1–M12 taxonomy. To install it for Claude Code:

cp -r skill/use-openrouter-safely ~/.claude/skills/

It then activates on its own whenever you're working with OpenRouter, or you can invoke it directly with /use-openrouter-safely. The guidance is plain Markdown — skill/use-openrouter-safely/SKILL.md is worth reading even if you don't use Claude Code.

Provenance: where every published number comes from

No statistic in this repo is hand-typed. Two files are inputs — the audit dataset (findings/survey.json, written by the multi-agent sweeps) and the endpoint snapshot (findings/provider_spread_reference.json, fetched from OpenRouter's API). Everything else is generated from them:

uv run scripts/fetch_provider_spread.py   # refetch the endpoint snapshot   ⚠️ see below
uv run scripts/set_safety_class.py        # classify rows -> regenerates survey.csv,
                                          #   stats.json, artifact/_data.json, index.html
uv run scripts/build_claims.py            # derive every prose-cited statistic -> claims.json
uv run scripts/make_summary.py            # findings/summary.md
uv run scripts/fetch_prior_work_sources.py      # re-verify every prior-work quotation at source
uv run scripts/make_prior_work.py               # reports/prior-work.md
uv run scripts/add_fingerprint_column.py  # join per-repo fingerprint verdicts onto the dataset
uv run scripts/make_fingerprints.py       # reports/detection-fingerprints.md
uv run scripts/fetch_provider_transparency.py   # re-verify every vendor quotation at source
uv run scripts/make_provider_transparency.py    # reports/provider-transparency.md
uv run --with cairosvg scripts/make_poster.py   # image/openrouter_findings.{svg,png}
uv run --with pytest pytest tests/        # enforces the whole chain

findings/claims.json is the ledger: each entry records a value, its source file, and the derivation used. Markdown cannot \input{} a value, so tests/test_claims_provenance.py is the compensating control — it asserts the numbers written in this README, findings/summary.md, and the skill still match the data. A statistic that drifts fails a test instead of quietly going stale.

Derived values are never persisted into the input snapshots, only into claims.json. (A quant_spread field once stored in the endpoint snapshot disagreed with its own endpoint list for 78 of 87 models; it was removed rather than repaired.)

⚠️ Refetching the endpoint snapshot moves published numbers. Endpoints rotate constantly. fetch_provider_spread.py will not reproduce the committed snapshot — rerun build_claims.py and the test suite afterwards and update whatever the tests flag. Treat it as a reviewed data change, not a routine refresh.

The two multi-agent sweeps (importance-first discovery → per-repo audit + adversarial verify) were run as background workflows; the discovery record is preserved in findings/discovered_candidates.json.