EvalHub

A lightweight REST API service for orchestrating LLM evaluations across multiple backends. Written in Go, it routes evaluation requests to frameworks like lm-evaluation-harness, RAGAS, Garak, and GuideLLM orchestrated via a complementary SDK, tracks experiments via MLflow, and runs natively on OpenShift.

Architecture

The service uses Go's standard net/http router, structured logging with zap, Prometheus metrics, and a pluggable storage layer (SQLite for development, PostgreSQL for production). Providers and benchmarks are declared in YAML configuration files shipped with the container image.

Quick start

Prerequisites

• Go 1.26+
• Make
• Python 3 (for make test; used by scripts/grcat for colored output)
• uv (manages the Python venv required by make start-service and FVT tests; run make venv to create it)
• Podman (for container builds)
• Access to an OpenShift or Kubernetes cluster (for deployment)

Run locally

make install-deps
make build
./bin/eval-hub

Note that in some cases it may be necessary to exclude certain (newer) dependencies, this can be done as shown below in the go.mod file:

exclude (
  k8s.io/api v0.36.0
)

The API is available at http://localhost:8080. Verify it is running:

curl http://localhost:8080/api/v1/health

Interactive documentation is served at /docs.

Run in a container

podman build -t eval-hub:latest -f Containerfile .
podman run --rm -p 8080:8080 eval-hub:latest

Deploy to OpenShift

EvalHub is managed by the TrustyAI Service Operator via a custom resource:

apiVersion: trustyai.opendatahub.io/v1alpha1
kind: EvalHub
metadata:
  name: evalhub
  namespace: my-namespace
spec:
  replicas: 1
  env:
    - name: MLFLOW_TRACKING_URI
      value: "http://mlflow:5000"
    - name: EVALHUB_HARDWARE_PROFILES_NAMESPACE
      value: "opendatahub"  # or redhat-ods-applications on RHOAI

Apply the CR to your cluster:

oc apply -f evalhub-cr.yaml
oc get evalhub -n my-namespace    # check status

Local development

make start-service          # start in background (logs to bin/service.log)
make stop-service           # stop

make test                   # unit tests
make test-fvt               # BDD functional tests (godog)
make test-all               # both
make test-coverage          # generate coverage.html

make lint                   # go vet
make fmt                    # go fmt

Run a single test:

go test -v ./internal/handlers -run TestHandleName

To create a Python wheel distribution of the server for local development and testing:

make cross-compile
make build-wheel

Exposing private functions for tests

Create a file called export_test.go in the package under test and re-export symbols needed by _test.go files in other packages.

Database

SQLite in-memory is the default (database.driver: sqlite in config/config.yaml). To use PostgreSQL locally there are two approaches: a container or a native install. Both use targets in tests/postgres/Makefile.

Note: The credentials and auth settings below are for local development and testing only. For production deployments, use strong passwords, TLS, and appropriate authentication mechanisms.

Option 1: Container (Podman/Docker)

No system-level install required. The container creates the database, user, and permissions automatically.

cd tests/postgres
POSTGRES_PASSWORD=<your-password> make start-postgres-container

To stop and remove:

cd tests/postgres
make stop-postgres-container
make delete-postgres-container

Configure EvalHub in config/config.yaml:

database:
  driver: pgx
  url: postgres://eval_hub:<your-password>@localhost:5432/eval_hub

Or override via environment variables:

export DB_DRIVER=pgx
export DB_URL="postgres://eval_hub:<your-password>@localhost:5432/eval_hub"

Option 2: Native install (Homebrew on macOS, apt on Linux)

cd tests/postgres
make install-postgres
make start-postgres
make create-user
make create-database
make grant-permissions

To stop:

cd tests/postgres
make stop-postgres

Configure EvalHub in config/config.yaml (no password needed with trust/peer auth):

database:
  driver: pgx
  url: postgres://eval_hub@localhost:5432/eval_hub

Or override via environment variables:

export DB_DRIVER=pgx
export DB_URL="postgres://eval_hub@localhost:5432/eval_hub"

Configuration

Configuration is loaded from config/config.yaml, overridden by environment variables and secret files.

Variable	Purpose	Default
PORT	API listen port	8080
DB_DRIVER	Database driver (sqlite or pgx)	sqlite
DB_URL	Database connection string	SQLite in-memory
MLFLOW_TRACKING_URI	MLflow tracking server	http://localhost:5000
MLFLOW_CA_CERT_PATH	PEM CA bundle for MLflow TLS verification	(system roots)
LOG_LEVEL	Logging level	INFO
EVALHUB_HARDWARE_PROFILES_NAMESPACE	Platform namespace where OpenDataHub HardwareProfile CRs are fetched (Kubernetes runtime). Required for hardware_config.hardware_profile_name evaluations; typically opendatahub or redhat-ods-applications. Set by the TrustyAI Service Operator deployment.	(unset — hardware profile lookups fail)

Provider configurations live in config/providers/ as YAML files. The default set includes lm-evaluation-harness (167 benchmarks), RAGAS, Garak, GuideLLM, LightEval, and MTEB.

Syncing providers and collections to the TrustyAI operator

Provider and collection definitions are maintained here and mirrored as ConfigMaps in the TrustyAI Service Operator:

• Providers: config/providers/ → config/configmaps/evalhub/provider-*.yaml
• Collections: config/collections/ → config/configmaps/evalhub/collection-*.yaml

When adding or changing a provider or collection, update the source YAML in this repository and the corresponding embedded ConfigMap in the operator repository. Add new ConfigMaps to the operator's config/configmaps/evalhub/kustomization.yaml. Keep the two repositories' changes coordinated so the operator can deploy the same definitions.

The sync check compares the embedded ConfigMap YAML with the source files. Run it locally with:

python scripts/check_configmap_sync.py

The same check runs in CI through the TrustyAI Operator ConfigMap Sync workflow.

API overview

All endpoints are versioned under /api/v1. Full specification at eval-hub.github.io/eval-hub.

Endpoint	Methods	Description
/api/v1/evaluations/jobs	POST, GET	Create or list evaluation jobs
/api/v1/evaluations/jobs/{id}	GET, DELETE	Get status or cancel a job
/api/v1/evaluations/collections	GET, POST	List or create benchmark collections
/api/v1/evaluations/providers	GET, POST	List or create providers
/api/v1/evaluations/providers/{id}	GET, PUT, PATCH, DELETE	Manage a provider
/api/v1/evaluations/jobs/{id}/events	POST	Submit job events
/api/v1/health	GET	Health check (no identity headers; no build/version fields)
/metrics	GET	Prometheus metrics

Detailed API documentation: eval-hub.github.io/eval-hub

Custom backends

EvalHub supports Bring Your Own Framework (BYOF). Extend the FrameworkAdapter class from the eval-hub-sdk and implement a single method -- EvalHub handles scheduling, status reporting, and result aggregation.

from evalhub.adapter import FrameworkAdapter, JobSpec, JobCallbacks, JobResults, EvaluationResult

class MyAdapter(FrameworkAdapter):
    def run_benchmark_job(self, config: JobSpec, callbacks: JobCallbacks) -> JobResults:
        # run your evaluation logic, report progress via callbacks
        callbacks.report_status(JobStatusUpdate(status=JobStatus.RUNNING, progress=0.5))
        score = evaluate(config.model, config.parameters)
        return JobResults(
            id=config.id,
            benchmark_id=config.benchmark_id,
            model_name=config.model.name,
            results=[EvaluationResult(metric_name="accuracy", metric_value=score)],
            num_examples_evaluated=100,
            duration_seconds=elapsed,
        )

Register the new provider by adding a YAML entry to the providers ConfigMap. No additional services or TCP listeners are required -- adapters run as jobs, not servers. Once registered, the provider and its benchmarks are available through the standard /api/v1/evaluations/providers endpoint.

Project structure

eval-hub/
├── cmd/eval_hub/          # Entry point (main binary)
├── internal/
│   ├── handlers/          # HTTP request handlers
│   ├── storage/           # Database abstraction (SQLite, PostgreSQL)
│   ├── mlflow/            # MLflow client
│   ├── runtimes/          # Backend execution adapters
│   ├── config/            # Viper-based configuration
│   ├── validation/        # Request validation
│   ├── metrics/           # Prometheus instrumentation
│   └── logging/           # Structured logging (zap)
├── config/                # config.yaml and provider definitions
├── docs/src/              # OpenAPI 3.1.0 specification (source of truth)
├── tests/features/        # BDD tests (godog)
├── Containerfile          # Multi-stage UBI9 container build
└── Makefile               # Build, test, and dev targets

Local mode

EvalHub can run evaluations locally without a Kubernetes cluster. See the local mode guide for configuration, architecture details, and troubleshooting, and the local mode tutorial for a step-by-step walkthrough. A self-contained LightEval example is included in this repository.

When local mode has mlflow.tracking_uri configured, each local evaluation subprocess automatically receives that direct URI as MLFLOW_TRACKING_URI. Local subprocess environment variables are applied in this order, with later values replacing matching earlier values:

1. Inherited process environment
2. EvalHub and service configuration values
3. Provider runtime.local.env values

Further reading

• API documentation -- full endpoint reference
• Local mode guide -- running evaluations without Kubernetes
• CONTRIBUTING.md -- contribution guidelines
• OpenAPI spec -- machine-readable API definition

Licence

Apache 2.0 -- see LICENSE.