feat: add OrcaRouter as a first-class API model provider

[nissrin2020ali-ux]

Motivation

OpenCompass calls itself "a compass for the complex landscape of evaluating large language models", and its API model ecosystem currently covers OpenAI, LiteLLM (as a unified gateway), ChatGLM, MiniMax, XunFei and others. LiteLLM already lets users reach many providers, but gateways such as OrcaRouter also expose their own provider/model namespace and value-add features that sit behind the OpenAI-compatible endpoint. Today an OpenCompass user who wants OrcaRouter has to treat it as an anonymous custom openai_api_base, losing the ergonomics of a first-class provider.

OrcaRouter is an OpenAI-compatible AI gateway built for both models and agents. Like OpenRouter, it exposes a provider/model namespace across many models — but it also combines adaptive routing, automatic failover, zero-markup inference, observability, guardrails, and agent-tool governance behind the same endpoint. Adding orcarouter as a first-class provider means this project's users can use that stack directly, without treating OrcaRouter as an anonymous custom base URL. It also runs gateway-level, zero-trust security for AI agents on the same endpoint — screening every prompt/response and governing every tool call on a default-deny basis, with no application code changes.

Modification

• Add opencompass/models/orcarouter_api.py with OrcaRouterAPI (registered) and OrcaRouterAPIStreaming (registered). Both mirror the existing OpenAISDK / OpenAISDKStreaming integrations (the same classes OpenCompass already uses for DeepSeek-R1 and similar OpenAI-compatible endpoints), defaulting to the OrcaRouter gateway base URL https://api.orcarouter.ai/v1/ and reading the key from the ORCAROUTER_API_KEY env var when key='ENV'.
• Export both classes from opencompass/models/__init__.py.
• Add tests/models/test_orcarouter_api.py covering registry registration, key/base defaults and the ORCAROUTER_API_KEY fallback, explicit key override, reasoning-content merging, and streaming chunk parsing.
• Add examples/eval_orcarouter.py demonstrating a minimal evaluation config.
• Document the provider in docs/en/user_guides/models.md under API-based Models.

Use cases

from opencompass.models import OrcaRouterAPI

models = [
    dict(
        abbr='OrcaRouter-Free',
        type=OrcaRouterAPI,
        path='orcarouter/free',   # any model id from the gateway
        key='ENV',                # $ORCAROUTER_API_KEY
        meta_template=api_meta_template,
        query_per_second=1,
        max_out_len=512,
        max_seq_len=16384,
        batch_size=1,
    ),
]

Verification

• pytest tests/models/test_orcarouter_api.py — 10/10 passed.
• pytest tests/models/test_openai_api.py tests/models/test_litellm_api.py — 45/45 passed (no regressions).
• flake8, isort, yapf, and codespell clean on all changed files.
• Live check against https://api.orcarouter.ai/v1 through OrcaRouterAPI.generate() returned a real completion (200), and the streaming variant streamed chunks back successfully.

Checklist

• Pre-commit or other linting tools are used to fix the potential lint issues.
• The modification is covered by complete unit tests.
• The documentation has been modified accordingly.

---

I'm an engineer on the OrcaRouter team.

Discord: discord.gg/YEubt8enRA · X: https://x.com/OrcaRouter