Skip to content

feat(eval): add optional API Route reader - #167

Open
DennyHo0917 wants to merge 1 commit into
StarTrail-org:mainfrom
DennyHo0917:feat/api-route-reader
Open

DennyHo0917 wants to merge 1 commit into
StarTrail-org:mainfrom
DennyHo0917:feat/api-route-reader

Conversation

@DennyHo0917

Copy link
Copy Markdown

Summary

Add an optional API Route reader to the evaluation CLI, following the existing Cheaper Inference (#163) and Atlas Cloud (#159) reader integrations.

  • Select explicitly with --api-route; the base URL is https://global.api-route.com/v1.
  • Resolve only --api-key or API_ROUTE_API_KEY, preserving exact account model IDs. Other providers' keys are never used.
  • Force the existing OpenAI-compatible client even for model IDs containing gemini.
  • Reject other provider flags, LiteLLM and endpoint overrides when this reader is selected.
  • Disable automatic request retries for this optional reader, as Atlas Cloud already does, to avoid duplicating billable generations.
  • Add setup documentation and offline regression tests. Existing reader defaults and dependencies remain unchanged.

Validation

  • pytest tests/test_api_route_client.py tests/test_atlascloud_client.py tests/test_cheaperinference_client.py -q: 29 passed.
  • Actual run_bench.py --help succeeds and exposes --api-route.
  • Actual CLI rejects all six competing provider flags and --api-base with exit code 2 before loading a dataset or generating an answer.
  • Ruff lint and format checks passed for the three affected Python files; git diff --check passed.
  • Windows validation used an isolated uv run --no-project --with ... environment because the project restricts its UV environments to Linux/macOS. No project dependency/lockfile change was needed.
  • HTTP/SDK calls are mocked in the tests. No live inference, vision-model compatibility or end-to-end retrieval benchmark was tested; the docs require users to verify image support for screenshot readers.

Disclosure

I operate API Route. This contribution was prepared with AI assistance and checked locally. It adds no model context/pricing/capability metadata or default model recommendation.

Signed-off-by: DennyHo0917 <149746199+DennyHo0917@users.noreply.github.com>
@vercel

vercel Bot commented Oct 1, 2026

Copy link
Copy Markdown

@DennyHo0917 is attempting to deploy a commit to the andylizf's projects Team on Vercel.

A member of the Team first needs to authorize it.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant