ACME Agents: metering and rebilling through the LangWatch AI Gateway

A small agent-platform SaaS, complete enough to be believable: customers sign up, create agents, chat with them, and watch what they are spending. The platform meters every LLM call and rebills its customers, and it does so with zero metering code of its own. Everything hard is delegated to LangWatch:

• Provisioning: signing up a customer mints one virtual key (the tenant boundary), a hard cap, a soft cap, and a per-seat allowance, through the official LangWatch SDKs (TypeScript and Python) with one org API key. Flip LANGWATCH_PROVISION_MODE to project and the same signup gives every customer a LangWatch project of its own first, with the key scoped to it, so their traces and costs land in a project only they are in.
• Request path: chat calls the gateway on the OpenAI wire with the user field set. That one field is all the attribution the whole billing pipeline needs.
• Billing: the meters on screen are fed by LangWatch's signed webhooks and its budget APIs. If the numbers are right, the pipeline works end to end.
• Reconciliation: aggregate checksums first, cursor diff only on divergence, and the same walk backfills whatever the books are missing.
• Period close: an explicit budget reset that never mutates recorded spend.

This repo is the canonical integration example our docs point at. If you are building the same shape of platform, point your coding agent at this repo and at the migration guide; every file is written to be copied from.

What it looks like

Sign-up provisions the tenant on LangWatch and drops straight into the dashboard, so the customer sees the outcome instead of a spinner:

The workspace: agents on the left, a streaming chat in the middle, and a usage meter underneath showing real spend against the real caps, with the signed billing events that produced those numbers below it.

When a cap is reached the customer gets a sentence, not a stack trace, and the message stays in the box so it can be retried. The wording follows the 402's error.meta.budget_scope: a personal allowance, a company cap on the key and a company cap on the customer's own project are different problems.

The SaaS owner console is the behind-the-scenes view: every customer, their caps and spend, per-seat allowances, the reset and cap actions, receiver health, and a live feed of billing events as they are delivered.

The developer panel keeps every raw envelope one click away, next to the two snippets that are the whole integration.

Layout

app/         the SaaS itself (:4100)
  src/       Express API, the gateway request path, the in-process webhook receiver
  web/       the browser app (React + Tailwind, built by Vite)
ts/          the integration surfaces in TypeScript (standalone receiver on :4101)
python/      the SAME integration surfaces in Python (receiver on :4102),
             plus the SaaS shell again as one FastAPI process (:4200)
scripts/     seed two fictional tenants, register both app shells' webhook
             endpoints, create the team customer projects go under

Both app shells consume the official LangWatch SDK for their language: zero hand-rolled HTTP and zero hand-rolled crypto. Provisioning, cap reads, budget resets and reconciliation are SDK calls, and so is the one piece of security-critical code an integration used to write itself: webhook signatures are verified by the SDK's own verifier, which takes every secret the receiver currently holds so a rotation never drops a delivery.

The integration surfaces exist twice, one per language, each self-contained:

Surface	TypeScript	Python
Webhook receiver	ts/src/receiver.ts	python/receiver.py
Billing ledger	ts/src/ledger.ts	python/ledger.py
Reconciliation	ts/src/reconcile.ts	python/reconcile.py
Provisioning	ts/src/provision.ts	python/provision.py
App shell	app/ (Express + React, :4100)	python/app.py (FastAPI, :4200)

The two app shells are split the same way, so a module can be read beside its twin: routes (app/src/server.ts / python/app.py), the gateway request path (gateway.ts / gateway.py), every platform call (langwatch.ts / platform_api.py), the meters (usage.ts / usage.py), the error envelope (errors.ts / errors.py), the SSE fan-out (events.ts / live_feed.py), the webhook archive (webhooks.ts / webhook_ingest.py), the schema (db.ts / store.py) and the money rules (money.ts / money.py).

The app also hosts its own receiver at app/src/webhooks.ts, which is what makes the meters move on their own: an ingested event is written, then pushed to the browser over Server-Sent Events in the same breath.

Running it

You need a LangWatch instance (local dev, self-hosted, or cloud) and an org API key with gateway permissions.

cp .env.example .env      # fill in LANGWATCH_API_KEY, LANGWATCH_PROJECT_ID and the URLs
pnpm install

pnpm setup:webhook        # registers both app shells' receivers, writes their secrets to .env
pnpm build                # builds the browser app into app/web/dist
pnpm dev                  # ACME Agents on http://localhost:4100

To provision a project per customer instead, run the one-time team setup and flip the mode:

pnpm setup:team           # creates the team customer projects go under, writes LANGWATCH_TEAM_ID
# then in .env: LANGWATCH_PROVISION_MODE=project

Open http://localhost:4100, create a workspace, add an agent, and chat. The usage meter fills from the billing events the gateway delivers to this same process.

Optional extras:

pnpm seed                 # two fictional tenants, already provisioned
pnpm dev:web              # Vite with hot reload on :4300, proxying the API to :4100
pnpm receiver:ts          # the standalone TypeScript receiver on :4101
python python/receiver.py # the standalone Python receiver on :4102

The same browser app, either backend

The two app shells implement the same HTTP contract on purpose: same paths, same request bodies, same response shapes, same status codes, and the same {"error": {"code", "message", "hint"}} envelope. The React app in app/web is one bundle, and it cannot tell which one it is talking to. Sign-up, the streaming chat, the live meters, the event feed, the owner console and the period close all work against either.

pnpm dev                  # TypeScript app on APP_PORT, serving the UI there too
pnpm dev:python           # Python app on APP_PY_PORT, serving the same UI there

Both ports come from .env (APP_PORT, default 4100; APP_PY_PORT, default 4200). Each shell prints the port it bound next to the URL it advertises to LangWatch, and refuses to start when the two disagree, because a receiver registered on one port and listening on another looks like a broken webhook rather than a wrong number.

Both serve the built bundle from app/web/dist, so each port is a complete app on its own. For hot reload, Vite runs on :4300 and proxies to whichever backend you point it at:

pnpm dev:web              # :4300 against the TypeScript app
pnpm dev:web:python       # :4300 against the Python app

Both read the shell's port out of .env, so there is no port to repeat. DEMO_API_TARGET still overrides the origin outright when the backend is somewhere else entirely, a tunnel or another host:

DEMO_API_TARGET=https://acme.ngrok.app pnpm dev:web

The two shells keep separate databases (app/app.sqlite and python/app_py.sqlite) and separate webhook endpoints, so a tenant signed up on one does not appear on the other. Everything they read back, the caps, the spend and the delivered billing events, comes from the same LangWatch organization: caps and spend are organization-wide reads, so they answer the same whether every tenant sits in one project or in a project each.

Reconciling

pnpm reconcile:ts         # or: python python/reconcile.py

Exit code 0 means every virtual key's local totals match LangWatch's checksums for the last 24 hours. Delete a row from ts/ledger.sqlite and run it again to watch the cursor diff name exactly the missing request id, pull it back from GET /api/gateway/v1/spend-events, write it into the local ledger, and re-read the checksum as reconciled.

Reconciliation repairs by PULLING, never by asking for a redelivery. A replayed envelope carries its original id, and every receiver dedups on ids forever, so replaying a window your receiver has already seen is a guaranteed no-op no matter what is missing from your books. POST /spend-events/replay is a redelivery test tool, for proving an endpoint receives and verifies what it is sent. It is not a repair.

Receiving on a queue instead of a URL

An endpoint can deliver to an Amazon SQS queue instead of an HTTPS route, and this repository implements both, in both languages. Set DEMO_TRANSPORT=sqs, give each receiver its own standard queue, and run setup again:

APP_QUEUE_URL=https://sqs.<region>.amazonaws.com/<account>/<queue>
APP_PY_QUEUE_URL=...   # each receiver needs its OWN queue: two consumers on
TS_QUEUE_URL=...       # one queue split the messages rather than both seeing
PY_QUEUE_URL=...       # everything

DEMO_TRANSPORT=sqs pnpm setup:webhook
DEMO_TRANSPORT=sqs pnpm dev            # the app drains its queue in-process
pnpm --filter @acme/integration-ts exec tsx src/queue-consumer.ts
cd python && .venv/bin/python queue_consumer.py

The one thing that catches every consumer: ReceiveMessage returns no message attributes unless you ask for them by name. Pass MessageAttributeNames: ["All"]. Forget it and the signature is simply not there, so your consumer rejects every message it is sent while the body in front of you looks perfectly fine. All four consumers here carry that line with a comment saying why.

Nothing else changes. The message body is byte-identical to the HTTP body, {"batch": [...]}, and the signature, the delivery id and the attempt ride as message attributes under the same names they use as HTTP headers, so the verification code is the same call over the same bytes. What differs is how you say "retry": delete the message only after the batch is durably ingested, and leave it alone otherwise. It returns after the visibility timeout and, after maxReceiveCount attempts, lands in the dead letter queue.

The edges worth trying

• Breach: chat until a cap trips, or lower one from the owner console. The UI distinguishes "your allowance" from "your company's cap" using the 402's error.meta.budget_scope, which arrives as attributed_user, virtual_key or project depending on the cap that ran out, and the message you tried to send is handed back for a retry.
• Period close: the owner console resets the manual-window caps. Traffic admits again, the ledger keeps every recorded row, and the month-window seat allowances are left to roll over on their own.
• Switch customer: the account menu swaps agents, transcripts, meter and event feed together. Nothing is shared between tenants but the code.
• Tamper: curl -X POST localhost:4100/webhooks/langwatch -d '{}' and watch the receiver reject the unsigned body with malformed_header. Change one byte of a signed body and it comes back invalid_signature; re-send a capture from an hour ago and it comes back stale_timestamp.

How a customer comes to exist

LANGWATCH_PROVISION_MODE decides what a signup creates. virtual_key, the default, mints one key per customer under LANGWATCH_PROJECT_ID and hangs every cap off that key. project creates the customer a LangWatch project of its own first, under the one team pnpm setup:team made, scopes the key to that project and points its traces there, and hangs the caps off the project. Either way LANGWATCH_PROJECT_ID stays the project these calls authenticate as: the customer's own project is data in the request bodies, never a change of who is calling.

sequenceDiagram
    participant Browser
    participant App as ACME Agents app
    participant LW as LangWatch REST
    participant GW as AI Gateway

    Browser->>App: POST /api/customers {name, email}
    alt provisioning mode is project
        App->>LW: GET /api/projects, matching the customer name
        App->>LW: POST /api/projects {name, teamId} when there is no match
        LW-->>App: {project.id} and a service key this app never stores
    end
    App->>LW: POST /api/gateway/v1/virtual-keys
    Note over App,LW: in project mode the key carries scopes<br/>and trace_project_id for that project
    LW-->>App: {virtual_key.id, secret}  (secret shown once)
    App->>LW: POST /budgets {tenant scope, manual, $5, block}
    App->>LW: POST /budgets {tenant scope, manual, $2.50, warn}
    App->>LW: POST /budgets {attributed_user, month, $1, block, cycle_anchor_at: now}
    App->>App: store {vk id, secret, budget ids, project id} on the customer row
    App-->>Browser: 201, redirect into the dashboard
    Note over App,GW: the tenant scope is the customer's project when it has one<br/>and its virtual key otherwise. Seats need NO provisioning,<br/>their buckets appear on first spend

Loading

Every one of those creates carries an idempotency key derived from the customer's name, so a double-submitted form or a retry after a timeout returns the SAME key and the SAME budgets rather than a second set, and the app says so instead of reporting a fresh provisioning. The monthly seat allowance carries a cycle_anchor_at of the signup instant, so the customer's billing period runs from the day they started rather than from the calendar first, and the usage meter shows the dates.

The project create is the one call with no idempotency key, so the name does that job: the app looks for a project of the customer's name before creating one, and a retried signup finds the project the first attempt made instead of stacking a second one beside it.

A name that is already taken never reaches the platform: the app checks first, answers 409 {"error": {"code": "customer_exists"}} with the existing workspace, and offers to open it. If two sign-ups race past that check, the key minted a moment earlier is revoked rather than left orphaned, and the project goes with it when this signup is the one that created it.

How money flows (and what happens at the edges)

sequenceDiagram
    participant App as ACME Agents app
    participant GW as AI Gateway
    participant LW as LangWatch
    participant RX as Webhook receiver
    participant UI as Usage meter

    App->>GW: POST /v1/chat/completions (VK secret, user: seat email)
    GW-->>App: streamed response + X-LangWatch-Gateway-Request-Id
    LW->>RX: POST batch {gateway.request.completed} signed t=,v1=[,v1=]
    RX->>RX: verify HMAC over raw body (5 min tolerance, any v1 may match)
    RX->>UI: ingest (dedup by envelope id), push over SSE
    Note over LW,RX: a request whose confirmation never arrived is delivered<br/>as gateway.request.settled with null cost. If the real completion<br/>arrives later it SUPERSEDES the settled row: replace, never sum

    App->>GW: chat until a cap is crossed
    GW-->>App: 402 {error: {code: budget_exceeded, meta: {budget_scope, budget_id, budget_window}}}
    LW->>RX: gateway.budget.threshold_crossed, then gateway.budget.breached
    Note over App: meta.budget_scope tells the app whether to say<br/>"your allowance ran out" (attributed_user)<br/>or "your company's cap ran out" (virtual_key, project)
    Note over LW,RX: budget events name their tenant directly:<br/>virtual_key_id and anchor_project_id are first-class

    App->>LW: POST /api/gateway/v1/budgets/:id/reset  (period close)
    Note over LW: reset moves the manual window boundary.<br/>Recorded spend and emitted events are immutable
    App->>GW: traffic admits again

Loading

The rules this code follows

Worth internalizing before you adapt it:

1. Verify the raw bytes with the SDK verifier. verifyWebhookSignature (verify_webhook_signature in python) covers the exact body received, so parse after verification and never before. Hand it every secret the receiver currently holds: v1 repeats while a secret is rotating, one per currently valid secret, and the delivery is good when any of them matches. It throws rather than returning false, so a delivery cannot be trusted by forgetting to read a return value, and the error's code says which check failed (malformed_header, stale_timestamp, invalid_signature).
2. Dedup by the envelope id. Delivery is at-least-once; ingest must be idempotent. Ids are stable across retries and replays. X-LangWatch-Delivery-Id names the DELIVERY, which carries a whole batch, so it is a log correlation handle and never the dedup key.
3. Completed supersedes settled. Replace, never sum, joined on gateway_request_id. This app picks the winner at read time, so both envelopes stay archived.
4. Money is integer nano-USD. Every row carries *_nano_usd beside the *_usd display string. Read and sum the integer; convert exactly once, at the edge, where a figure becomes something a person looks at. Never parse a float out of the string. A spend figure the platform could not total arrives as null and must stay null: rendering it as $0.00 is a number the reader takes for real money.
5. Show the figure the platform enforces. Caps and their spend are read back from LangWatch, so a seat can never sit at "37% used" on screen while the gateway is already refusing its requests.
6. Checksums before walks, and pull to repair. One spend-summaries call reconciles the common case; the cursor walk is only for divergence, and it carries the full envelopes, so it also repairs. Never reach for a replay to fix your books.
7. Answer 2xx only after durable ingest. A failed database write should fail the delivery so LangWatch retries it.
8. One error shape, read once. Every LangWatch refusal crosses the wire as {"error": {"type", "code", "message", "meta"?}}, where type and code carry the same value and meta holds the machine-readable detail for that code. Read that shape; do not write a parser that tolerates several. This app's own API answers its browser in its own shape, {"error": {"code", "message", "hint"}}, so a person always gets a sentence they can act on and a driver error never leaks.
9. Enums are lowercase snake, in and out. Scope kinds (virtual_key, attributed_user, organization, project, principal, group, team), windows (manual, month, ...), on_breach (block, warn) and virtual key status (active, disabled, revoked). Uppercase is rejected on create, so there is one spelling to send and one to match on. The webhook ENDPOINT status is a separate enum on a separate surface and is still ACTIVE / DISABLED.
10. Spend windows are epoch milliseconds. from and to on every spend route are integer milliseconds, not ISO strings.

License

MIT. All tenant names fictional.