llama-dash turns a self-hosted local inference box into an observable, policy-controlled AI gateway: one UI for model state, request history, API keys, routing rules, proxy metrics, and client setup. The implemented inference backend is currently llama-swap over llama.cpp.
It is the single public entrypoint for OpenAI-compatible and Anthropic-compatible clients. llama-dash owns proxy policy, logging, auth, routing, and backend normalization, your selected inference backend owns local model processes and inference when traffic is routed to local models.
OpenAI SDK / Claude Code / Continue / Open WebUI
│
▼
llama-dash :3000
dashboard · auth · logs · routing · metrics
│ │
▼ ▼
llama-swap :8080 direct /v1 upstreams
llama.cpp models · peers OpenAI · Anthropic
- Monitor traffic, token usage, models, and CPU/GPU health in one dashboard.
- Browse request history with filters, timing, attribution, token counts, and cost estimates. Usage rollups persist beyond raw request retention.
- Load and unload models, inspect their capabilities and history, and edit validated llama-swap configuration.
- Proxy OpenAI and Anthropic clients to local models, peers, or direct upstreams; native llama.cpp and llama-swap audio endpoints are supported too.
- Control access with dashboard login, API keys, rate limits, model permissions, and MCP relay allow-lists.
- Route or reject requests, rewrite models, inject credentials and system prompts, and set request size limits.
- Test chat, image, speech, and transcription models in the playground.
- View logs, manage retention, and export Prometheus metrics at
/metrics. - Navigate with keyboard shortcuts, including
g dfor the dashboard andg rfor requests.
|
Dashboard Live traffic, tokens, model residency, upstream and GPU health |
Playground Chat against local endpoints with request/response inspection |
Request detail Routing, attribution, latency, tokens, and payload metadata |
Logs Raw llama-swap, proxy, and upstream streams in one viewer |
|
|
|
|
|
|
Model detail Load history, stats, recent requests, and config context |
Speech playground Read any article and audio testing |
Policies Aliases, routing rules, passthrough auth, and request limits |
Requests Searchable history with filters, sorting, and histogram |
|
|
|
|
Choose the compose file that matches your GPU vendor. Both setups pull the published ghcr.io/ndom91/llama-dash:latest image, use ./config/config.yaml for llama-swap config, mount ./models/ for model files, and expose llama-dash on http://localhost:3000.
Clone the repository for its Compose files and example configuration, then create the local environment and llama-swap config:
git clone https://github.com/ndom91/llama-dash.git
cd llama-dash
cp .env.example .env
cp config/config.example.yaml config/config.yamlSet BETTER_AUTH_SECRET in .env to the output of openssl rand -base64 33. The compose files load .env via env_file and won't start without it. CREDENTIAL_ENCRYPTION_KEY is only needed if you later store upstream provider credentials.
Place a GGUF model in ./models/, then set the matching path and model ID in config/config.yaml. The supplied example expects ./models/llama-3.gguf and declares it as llama3.
docker compose -f docker-compose.amd.yaml pull
docker compose -f docker-compose.amd.yaml up -ddocker-compose.amd.yaml runs llama-swap's unified ghcr.io/mostlygeek/llama-swap:unified-vulkan image, passes through /dev/dri, and also mounts /dev/dri into llama-dash so AMD GPU stats work in the dashboard. The config directory is mounted into both services so llama-dash can atomically save config.yaml and llama-swap can reload it through -watch-config.
docker compose -f docker-compose.nvidia.yaml pull
docker compose -f docker-compose.nvidia.yaml up -ddocker-compose.nvidia.yaml runs llama-swap's unified ghcr.io/mostlygeek/llama-swap:unified-cuda13 image and requests gpus: all for the llama-swap service. This requires the NVIDIA Container Toolkit on the host. unified-cuda13 supports Turing (RTX 20xx, GTX 16xx) and newer GPUs; for Pascal cards such as the GTX 10xx or P40, change the image to unified-cuda. The config directory is mounted into both services so llama-dash can atomically save config.yaml and llama-swap can reload it through -watch-config.
Set LLAMA_DASH_IMAGE_TAG to a published image tag, such as a commit SHA tag, to pin llama-dash to a specific build. Contributors can build locally instead by adding -f docker-compose.build.yaml and --build to either command, for example:
docker compose -f docker-compose.amd.yaml -f docker-compose.build.yaml up -d --buildOpen http://localhost:3000, create the first dashboard user, then load your configured model from Models. llama-dash runs database migrations automatically when it starts.
- Node 24+
- pnpm
- A reachable llama-swap instance
cp .env.example .env # edit INFERENCE_BASE_URL to point at your instance
pnpm install
pnpm db:migrate # creates data/dash.db
pnpm dev # http://localhost:5173Copy .env.example to .env and fill in the values.
See the environment variables reference for the complete list of variables, defaults, and deployment notes.
Route any Anthropic SDK, including Claude Code, through llama-dash for logging, filtering, per-request inspection, and Anthropic subscription passthrough.
See the Claude Code & Anthropic passthrough guide for client setup, routing rules, subscription OAuth, and provider-key flows. For remote MCP servers, see the MCP relays guide.
On passthrough routes the client's Authorization header belongs to the
upstream provider. To attribute those requests to a llama-dash API key, also send
the key in x-llama-dash-api-key. For Claude Code, set
"ANTHROPIC_CUSTOM_HEADERS": "x-llama-dash-api-key: sk-..." in env. The header
is never forwarded upstream.
This project was developed with significant assistance from LLMs. Architecture decisions, implementation, and documentation were all shaped through human-AI collaboration.
MIT