You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Sentry KILOCODE-WEB-28HE: new error in production #9065
The issue is a real, recurring production failure reported from a deliberate instrumentation point, not user noise. Sentry KILOCODE-WEB-28HE is `captureException` in packages/web-shared/src/lib/ai-gateway/usage-record-client.ts:143 (tags.source = recordUsageInPrimaryRegion) after the SFO AI-gateway instance's usage-write handoff to the Frankfurt writer endpoint returned an unusable response. The event's tag `reason: http_500` shows the writer returned HTTP 500. The writer route (apps/web/src/app/api/internal/usage/record/route.ts) is contractually supposed to answer 200 with status recorded/duplicate/not_recorded, or 400 on schema failure; it has no try/catch, so a 500 means something threw out of the handler. Sentry shows exactly that: unhandled (error.handled=false) DatabaseErrors at transaction `POST /api/internal/usage/record` attributed by frame to `isFirstUsage` in processUsage.ts (timeout exceeded when trying to connect, connection terminated due to connection timeout, no more connections allowed) — 924 events in 7 days. Those unhandled throws become the HTTP 500 the handoff client treats as terminal, so it stops retrying and falls back to a local write from SFO. Failure consequence is bounded (billing record is still written, client request still returns 200) but the fallback re-adds the transatlantic row-lock hold and primary pool pressure that the writer endpoint exists to remove, at roughly 1000 handoff failures per week.
Recommended next step: associate duplicate. Recommend associating ＃9065 with ＃9032 (KILOCODE-WEB-2860, Sentry anaconda-nq/7781631899). Both are the same mechanism: captureException in the SFO region-handoff client after the Frankfurt writer returned HTTP 500, and Sentry only separated them into distinct groups because the release changed. Consolidating keeps one work item for a failure that already spans 7 Sentry groups over 29 days. The evidence chain in this triage is directly reusable by that item: the writer route has no try/catch and so answers 500 only for an unhandled throw; the unguarded `isFirstUsage` primary query in the writer's write-path closure is the step that can throw, and Sentry attributes 924 unhandled connection DatabaseErrors in 7 days to that frame at the same endpoint while ~1001 handoff failures were recorded — after which the client correctly declines to retry a 500 and falls back to a local SFO write. If the duplicate association is declined, the same chain supports a single planning item for two linked changes: make the writer endpoint honour its in-band outcome contract instead of returning 500 for an unhandled database error, and reduce the primary connection scarcity (per-instance pool cap and Supavisor limits) that drives those throws, given the unresolved pool-waiting versus event-loop question documented in usage-record-diagnostics.ts.
Same mechanism: the SFO AI-gateway usage-write handoff to the Frankfurt writer (/api/internal/usage/record) returns HTTP 500 because an unhandled DatabaseError escapes outside insertUsageRecord catch, so the client treats 500 as terminal and falls back to a local write. Sentry split these into separate groups only because the release changed. #9032 is the canonical issue.
Activity
kfab-cloud commentedon Oct 28, 2026
Kfab triage
Provisional classification: suspected bug.
The issue is a real, recurring production failure reported from a deliberate instrumentation point, not user noise. Sentry KILOCODE-WEB-28HE is `captureException` in packages/web-shared/src/lib/ai-gateway/usage-record-client.ts:143 (tags.source = recordUsageInPrimaryRegion) after the SFO AI-gateway instance's usage-write handoff to the Frankfurt writer endpoint returned an unusable response. The event's tag `reason: http_500` shows the writer returned HTTP 500. The writer route (apps/web/src/app/api/internal/usage/record/route.ts) is contractually supposed to answer 200 with status recorded/duplicate/not_recorded, or 400 on schema failure; it has no try/catch, so a 500 means something threw out of the handler. Sentry shows exactly that: unhandled (error.handled=false) DatabaseErrors at transaction `POST /api/internal/usage/record` attributed by frame to `isFirstUsage` in processUsage.ts (timeout exceeded when trying to connect, connection terminated due to connection timeout, no more connections allowed) — 924 events in 7 days. Those unhandled throws become the HTTP 500 the handoff client treats as terminal, so it stops retrying and falls back to a local write from SFO. Failure consequence is bounded (billing record is still written, client request still returns 200) but the fallback re-adds the transatlantic row-lock hold and primary pool pressure that the writer endpoint exists to remove, at roughly 1000 handoff failures per week.
Recommended next step: associate duplicate. Recommend associating ＃9065 with ＃9032 (KILOCODE-WEB-2860, Sentry anaconda-nq/7781631899). Both are the same mechanism: captureException in the SFO region-handoff client after the Frankfurt writer returned HTTP 500, and Sentry only separated them into distinct groups because the release changed. Consolidating keeps one work item for a failure that already spans 7 Sentry groups over 29 days. The evidence chain in this triage is directly reusable by that item: the writer route has no try/catch and so answers 500 only for an unhandled throw; the unguarded `isFirstUsage` primary query in the writer's write-path closure is the step that can throw, and Sentry attributes 924 unhandled connection DatabaseErrors in 7 days to that frame at the same endpoint while ~1001 handoff failures were recorded — after which the client correctly declines to retry a 500 and falls back to a local SFO write. If the duplicate association is declined, the same chain supports a single planning item for two linked changes: make the writer endpoint honour its in-band outcome contract instead of returning 500 for an unhandled database error, and reduce the primary connection scarcity (per-instance pool cap and Supavisor limits) that drives those throws, given the unresolved pool-waiting versus event-loop question documented in usage-record-diagnostics.ts.
Sources:
Triage is complete. This recommendation does not start planning or implementation. Findings and hypotheses are available in the Kfab work item.
pandemicsyn commentedon Oct 28, 2026
Merging into #9032 as a duplicate.
Same mechanism: the SFO AI-gateway usage-write handoff to the Frankfurt writer (
/api/internal/usage/record) returns HTTP 500 because an unhandledDatabaseErrorescapes outsideinsertUsageRecordcatch, so the client treats 500 as terminal and falls back to a local write. Sentry split these into separate groups only because the release changed. #9032 is the canonical issue.