Skip to content

feat(eval): add optional Cheaper Inference reader - #163

Merged
ASuresh0524 merged 1 commit into
StarTrail-org:mainfrom
aiapienthusiast:cheaperinference-provider
Oct 1, 2026
Merged

ASuresh0524 merged 1 commit into
StarTrail-org:mainfrom
aiapienthusiast:cheaperinference-provider

Conversation

@aiapienthusiast

Copy link
Copy Markdown
Contributor

This PR adds an optional --cheaperinference reader to eval/run_bench.py.
It uses the same pattern as #159.

Cheaper Inference is an OpenAI-compatible LLM gateway.
gpt-5.4 and gpt-5.4-mini accept image input.

Cheaper Inference is one of the fastest-growing AI routers.
Each model costs 15–60% less than the list price of its lab.

Details

Behavior:

  • Uses the existing OpenAI-compatible client.
  • Reads the key only from --api-key or CHEAPER_INFERENCE_API_KEY.
  • Sends model IDs unchanged.
  • Stops with an error when combined with another provider flag, --litellm or --api-base.

Files:

  • eval/lib/model_config.py
  • eval/run_bench.py
  • eval/README.md
  • tests/test_cheaperinference_client.py (new)

Tests:

  • ruff check . and ruff format --check .: passed.
  • New tests and model config tests: passed.

@vercel

vercel Bot commented Sep 29, 2026

Copy link
Copy Markdown

@aiapienthusiast is attempting to deploy a commit to the andylizf's projects Team on Vercel.

A member of the Team first needs to authorize it.

@ASuresh0524
ASuresh0524 merged commit 9106cf4 into StarTrail-org:main Oct 1, 2026
4 of 5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants