fix(gateway): meter Anthropic streaming input tokens from message_start

[haiderysfzi]

What changes

• Merge message_start.message.usage into the Anthropic streaming usage sources, so input and cache tokens are metered alongside the output tokens message_delta already reported.
• Merge the three usage sources separately instead of spreading them into one object, so a later output-only frame cannot erase input tokens an earlier frame reported.
• Assert input tokens, request-row cost, and the spend counter in the existing streaming integration test, and add unit coverage for valid, split-chunk, malformed, and non-streamed frames.

Why

Closes #54.

Anthropic reports streamed input and cache usage under message_start.message.usage, while UsageTee read only the top-level usage and delta.usage. Streamed input tokens never reached metering, so with the existing 1M-input/1M-output fixture the gateway recorded 1500 cents against the 1800 cents owed. That undercounts the llm_requests row and the spend_counters row a hard budget enforces against, so a project can outspend its cap on streamed traffic.

The gap survived because the streaming test asserted only outputTokens. It now asserts the input tokens, the row cost, and the counter, so an undercount fails the suite rather than the invoice.

Verification

• pnpm verify passes locally — it reached the critical integration step and stopped on API timeout flakes unrelated to this change, detailed below.
• Behaviour verified beyond the test suite — reverted usage.ts alone and re-ran the new unit file: 4 of 5 tests fail with input tokens at 0, and the single pass is the non-streamed top-level usage case the change does not touch. Restored, all pass. The integration assertion moves 1500 → 1800 cents on both the request row and the spend counter.
• Documentation updated, or no user-facing change — no user-facing surface changes; the correction is to recorded usage.

Commands run:

• pnpm --filter @facility/gateway test — 57/57 passed against an isolated facility_gw
• pnpm --filter @facility/gateway typecheck
• pnpm lint
• pnpm guards — 2 guards ran, 0 failed
• git diff --check

On the pnpm verify box: the failures are in services/api, which does not depend on @facility/gateway and cannot import the changed file. The gateway suite also runs after the API suite, against a separate database. Running the same 42-file API suite three times gave three different results: test/assistant.test.ts failed during pnpm verify, unmodified main passed 42/42, and this branch produced two 5-second timeouts in test/watchtower.test.ts. That is the same class of unrelated 5-second API timeout reported in #143. Happy to re-run anything on request.

Acceptance criteria from #54: message_start input and cache fields are metered; a later message_delta merges output without erasing them; unit coverage spans valid, split-chunk, and malformed frames; and the integration proof runs against local Postgres and a stub provider, with no live credentials or network.

[jungle-lethanh]

I followed the EOF case separately from this fix. #148 looks good for the message_start and merge changes, but there’s one edge case left in the shared SSE path.
If the last usage frame contains valid JSON but the stream closes before the final blank line, the response still passes through, but that usage never gets counted.
This looks separate from #54. Since SSE normally treats that last frame as incomplete, I’m wondering which behavior Facility actually wants here: follow the protocol strictly, or still recover the usage because it affects budget tracking.
I can add a focused test once that behavior is clear.

[haiderysfzi]

Good catch, and I reproduced it. With the last frame valid but unterminated, the bytes still pass through byte-exact and usage comes back with outputTokens: 0. The cause is in parsePending: the trailing remainder is handed to parseJsonText, so it gets JSON.parsed with the event:/data: prefix still attached and throws. parseSseFrame is never applied to the final partial frame. It predates this PR and affects the OpenAI path the same way, since that part of the tee is provider-agnostic.

On which behaviour Facility wants, I would argue for recovering it.

The repo already takes that position elsewhere: in writeMetering, when a request errored and providerMayHaveCharged with a measured cost of zero, the cost falls back to the estimate. That is the same judgement, which is that when the provider may have billed you count something rather than nothing.

Strict SSE framing governs what you forward, and the tee already forwards untouched, so counting a frame it fully received breaks no contract with the caller. UsageTee is a meter rather than an EventSource client.

And the consequence is not a cosmetic metric: spend_counters is what hardBudgetBlock enforces against, so a dropped usage frame is free tokens against a hard cap, and a truncated stream is exactly when you least want that. It stays conservative either way, since only valid JSON recovers and a genuinely truncated body still parses to nothing and counts nothing.

The change is small: route the flush remainder through parseSseFrame when it contains data: lines, falling back to parseJsonText otherwise. Happy to add it here if you would rather have one round, but I would lean toward keeping it out of #54 so this diff stays about the message_start source. Tell me which you prefer and I will push it, or leave it to you if you already have the test written.

[adrian-lorenzo]

Thanks for the solid contribution!

[haiderysfzi]

Thanks! @adrian-lorenzo. Happy to dig into the codebase. CI workflows still need a maintainer to approve the run since it's my first contribution, could someone hit the approve button when you get a chance?

[ophiocus]

Not asking to reopen anything — the production fix is right and the approval looks right to me. This is about the test corpus, and it's a follow-up rather than a blocker.

api.anthropic.com puts usage at the top level of message_delta, as a sibling of delta; delta carries only the stop fields. That's typed in this repo's own dependency — @anthropic-ai/sdk@0.112.5, resources/messages/messages.d.ts:

export interface RawMessageDeltaEvent {
    delta: RawMessageDeltaEvent.Delta;   // stop_reason, stop_sequence, …
    type: 'message_delta';
    usage: MessageDeltaUsage;            // <- sibling, not inside delta
}

Two places still assume the nested shape. One is anthropicSseBody(), which predates this PR — it's been that way since the original gateway commit, so I've sent a separate PR for it against main (#181) rather than adding to this branch. The other is the comment added here in usage.ts:

message_delta carries output tokens under delta.usage

That one's worth a two-word edit before it lands, since it's the explanation the next person will read.

Why it matters beyond tidiness — two mutations to usage.ts on this branch, running test/usage.test.ts:

mutation	with these fixtures
drop parsed?.delta (dead against the real wire)	4 of 5 fail
drop parsed (the source that meters every streamed output token)	4 of 5 still pass

So the streaming tests are load-bearing on a frame that never arrives, and the source that actually meters streamed output is pinned only by the non-streamed case. mergeUsage's last-wins handles the real shape correctly already — there's just nothing holding it in place.

One practical note: I checked that my stub PR and this branch don't collide. Merged together locally, the gateway suite is 57/57, and your new inputTokens / costCents / spentCents assertions all still pass. If it's easier, ignore mine and fold the stub change in here instead — happy either way.

I also have the fixture corrections for usage.test.ts plus a cumulative-restatement case if you want them, but that's your call, not a request.