feat: decode inline Arrow IPC + warehouse-compat fallback

[jamesbroadhead]

Summary

Some warehouses return ARROW_STREAM + INLINE results as base64 Arrow IPC in result.attachment rather than result.data_array. Some refuse JSON_ARRAY + INLINE entirely and only support ARROW_STREAM for INLINE. Others (most classic + some serverless) do the opposite — they refuse ARROW_STREAM + INLINE and require EXTERNAL_LINKS. The previous code path silently returned empty results in all of these cases.

This PR makes the analytics plugin work across all three warehouse shapes by:

• handling both INLINE and EXTERNAL_LINKS dispositions for ARROW_STREAM,
• stashing inline Arrow IPC server-side and serving it via the existing /arrow-result route,
• falling back between dispositions based on what the warehouse signals it accepts,
• preserving the caller's requested format end-to-end (a JSON_ARRAY caller always gets JSON rows; an ARROW_STREAM caller always gets Arrow bytes), regardless of which disposition the warehouse actually accepted.

Design

Earlier iterations of this PR sent inline Arrow bytes over SSE (base64 inside JSON inside SSE framing) using an arrow_inline message type. That was reworked: inline payloads are now stashed server-side and delivered through the same /arrow-result/<jobId> endpoint that already serves EXTERNAL_LINKS results. The SSE control channel only ever carries { type: "arrow", statement_id }; the id is either a real warehouse statement id or a synthetic inline-<uuid>, and the server demultiplexes.

Wins from the redesign:

• Keeps multi-MiB blobs off SSE
• One client code path for both INLINE and EXTERNAL_LINKS
• Real binary content-type (application/vnd.apache.arrow.stream) instead of base64-in-JSON-in-SSE
• The retired arrow_inline wire message is gone — the discriminated union rejects it as schema-invalid

Stash properties (see inline-arrow-stash.ts):

• Drain-on-read (no replay path)
• TTL-bounded, evicted on put/take
• Per-user keyed
• Memory-bounded with rejection on overflow — callers fall back to EXTERNAL_LINKS rather than evicting in-flight entries
• Caveat: process-local. Multi-replica deployments need sticky sessions or a shared store (out of scope)

Disposition fallback

The plugin classifies an inline rejection into one of two signals (case-insensitive, requires INVALID_PARAMETER_VALUE / NOT_IMPLEMENTED plus mention of "inline" — unrelated errors aren't reinterpreted as disposition mismatches):

• needs-arrow: warehouse says it only accepts ARROW_STREAM for INLINE ("Inline disposition only supports ARROW_STREAM format."). If the caller asked for JSON_ARRAY, retry as ARROW_STREAM + INLINE and decode the Arrow IPC attachment to plain row objects server-side — the caller's JSON contract is preserved. Scalar values are stringified to match the warehouse's native JSON_ARRAY shape.
• needs-json: warehouse says it only accepts JSON_ARRAY for INLINE ("The format field must be JSON_ARRAY when the disposition field is INLINE.", "ARROW_STREAM is not supported with INLINE disposition"). If the caller asked for ARROW_STREAM, retry as ARROW_STREAM + EXTERNAL_LINKS.

For ARROW_STREAM callers, the stash-full case also falls back to EXTERNAL_LINKS. Explicit format requests are honored — no auto-downgrade across format boundaries (a JSON caller never gets Arrow bytes; an Arrow caller never gets JSON rows).

Changes

• Server-side Arrow IPC decoding (connectors/sql-warehouse/arrow-schema.ts + client.ts): detects result.attachment and decodes via apache-arrow's tableFromIPC. apache-arrow@21.1.0 added as a server dependency.
• Inline Arrow stash (plugins/analytics/inline-arrow-stash.ts): the bounded, TTL'd, drain-on-read, per-user keyed store described above.
• Unified /arrow-result route (plugins/analytics/analytics.ts): single endpoint serves both warehouse statement ids and synthetic inline-* ids; differentiation is by id prefix.
• Disposition classifier + bidirectional fallback (plugins/analytics/analytics.ts): replaces the prior heuristic, which case-sensitively required both INLINE and ARROW_STREAM in the error message and never matched the actual wire wordings.
• JSON_ARRAY server-side Arrow→rows decoder: when retrying with ARROW_STREAM, decode and stringify so the SSE payload still looks like {type: "result", data: [...rows]}.
• Zod-validated SSE wire protocol (shared/src/sse/analytics.ts): single source of truth between server and client. Default format remains JSON_ARRAY.

Tests

• connectors/sql-warehouse/tests/arrow-schema.test.ts (new, 514 lines)
• connectors/sql-warehouse/tests/client.test.ts (new, 383 lines) — real base64 Arrow IPC captured from a live serverless warehouse
• plugins/analytics/tests/inline-arrow-stash.test.ts (new, 116 lines)
• plugins/analytics/tests/analytics.test.ts (+770) — integration coverage for both fallback directions (warehouse refuses JSON_ARRAY + INLINE → server retries as ARROW_STREAM + INLINE and decodes; warehouse refuses ARROW_STREAM + INLINE → falls back to EXTERNAL_LINKS), plus stash-full fallback and the no-fallback safety net
• appkit-ui/src/react/hooks/__tests__/use-analytics-query.test.ts — arrow-IPC SSE tests (this PR) merged with the stable-params tests already on main from fix(appkit-ui): stabilize useAnalyticsQuery params reference to avoid infinite refetch #321
• shared/src/sse/analytics.test.ts (new, 87 lines)

Full suite: 2,674 tests, all green on the current head.

Stack

This was the third PR in a stack; both predecessors are now on main:

• test: backfill coverage for genie connector, service context, stream registry #327 — coverage backfill (merged)
• feat: rename AnalyticsFormat to API enum names with legacy aliases #328 — AnalyticsFormat rename to API enum names (merged)
• (this PR) — the actual fix

Test plan

Verified via automated tests on the current head:

• useAnalyticsQuery reads ARROW_STREAM results via /arrow-result for both real warehouse statement ids and synthetic inline-* ids
• JSON_ARRAY direct path: warehouse accepts INLINE + JSON_ARRAY → row data returned unchanged
• JSON_ARRAY fallback path: warehouse refuses INLINE + JSON_ARRAY but accepts INLINE + ARROW_STREAM → server decodes Arrow attachment to JSON rows; SSE wire carries {type: "result"}, not {type: "arrow"}
• JSON_ARRAY no-fallback safety: rejection without a needs-arrow signal does NOT trigger a retry
• ARROW_STREAM + INLINE → EXTERNAL_LINKS fallback when the warehouse rejects INLINE
• ARROW_STREAM + INLINE → EXTERNAL_LINKS fallback when the inline stash is full
• Inline-stash invariants: drain-on-read, TTL eviction, per-user keying, reject-on-overflow leaves prior entries intact
• Retired arrow_inline SSE message is rejected as schema-invalid and never triggers a local decode

Live smoke test against three e2-dogfood warehouses (SELECT 1 AS one, 'hello' UNION ALL SELECT 2, 'world'):

Warehouse	JSON_ARRAY caller (default)	ARROW_STREAM caller
serverless A (refuses JSON_ARRAY + INLINE)	✅ rejected on first attempt, fallback retries as ARROW_STREAM + INLINE, decoded to [{"one":"1","greeting":"hello"},{"one":"2","greeting":"world"}]	✅ INLINE + ARROW_STREAM accepted directly, attachment decoded
serverless B	✅ INLINE + JSON_ARRAY accepted directly	✅ INLINE rejected with needs-json, falls back to EXTERNAL_LINKS
classic	✅ INLINE + JSON_ARRAY accepted directly	✅ INLINE rejected with needs-json, falls back to EXTERNAL_LINKS

All three warehouses now return identical row shapes for the same default useAnalyticsQuery call.

Fixes #242. Replaces #256.

This pull request was AI-assisted by Isaac.