Repository navigation
feat(server): add experimental GET /api/experimental/media/:hash and opt-in experimental-media-uri capability - #59103
Conversation
…l-media-uri Clients that send X-OpenCode-Capabilities: experimental-media-uri get user files and tool results stored in blob as media:<hash> refs with their size and dimensions instead of inline base64, from session.messages, session.message and session.inbox.list, when experimental.media_blob is on in the session's own location (the config its media was written under). Refs whose blob is gone become the same unavailable placeholder inline readers get. Every other reader, the runner and export stay inline. GET /api/experimental/media/:hash serves one blob that a stored message or inbox item still references, with an immutable ETag, 304 support, nosniff and a sandbox CSP; anything but a raster image is sent as an application/octet-stream attachment. session.prompt accepts a referenced media:<hash> ref as an attachment URI and re-admits its stored bytes as is.
There was a problem hiding this comment.
I found nothing that blocks a merge, and the change works as described. The new server tests run the real server through ServerFetch and the generated client, both with and without the experimental-media-uri header and with experimental.media_blob on and off. They pass, as does session-owned.test.ts, and typecheck:tsc is clean for core, server and client.
Public API (additive, backwards compatible):
- New
GET /api/experimental/media/:hash(media.get, tagexperimental; promise clientexperimental.media.get, left out of the effect client). - New opt-in request header
X-OpenCode-Capabilities: experimental-media-uri. It changessession.messages,session.messageandsession.inbox.listonly when the session's location also hasexperimental.media_blobon. - New schemas:
Prompt.MediaRef,Prompt.MediaFileAttachment,Prompt.Attachment,Session.Message.MediaUser/MediaInfo,Session.Inbox.MediaUser/MediaUserPayload/MediaInfo. - Prompt attachments now also accept a
media:<hash>URI. server.experimentalnow holds both the persistent PTY routes and the media route.- Clients that don't send the header see no change.
Worth a look:
- With refs on, file items in assistant tool output also come back as
media:<hash>.describeonly addsbytes/width/heightto user and inbox files, so tool items have no size, and the spec still typesTool.Content.urias a plain string. Opted-in clients have to handle that case without it being documented. test/media.test.tsis about 360 lines for 2 tests, each a long chain of checks. Splitting it into a few small cases would make failures easier to locate: ref vs. inline shape, headers/304, the missing-blob placeholder, and the unreferenced-blob 404 with the rejected prompt.SessionBlob.splitis a new export that only the test uses here, so it could move to the backfill PR (#58742).
There was a problem hiding this comment.
The new commit changes only the test. Each test now takes its temp directory from tmpdirScoped before starting the servers, so the servers close the database before the directory is removed (Windows can't delete an open database). The assertions are the same as before. On the new head, bun test test/media.test.ts passes (2 tests) and server typecheck:tsc is clean. Production code and the public API are unchanged since my last review, and I have nothing new to raise.
There was a problem hiding this comment.
The new commit is sound. Opted-in reads now only offer a media: ref for a blob that something still references, using the same check the media route uses. Any other stored blob reads as the [media unavailable: …] placeholder instead of a ref that would 404, and the added test checks both the 404 and the placeholder. media.test.ts passes (2 tests), typecheck:tsc is clean in core and server, and re-running the protocol and client generators gives no diff.
Public API: the OpenAPI operationId changes from media.get to experimental.media.get. The route is new in this PR, so nothing released depends on the old id, and the generated client already calls it experimental.media.get. Nothing else changed. I have no new concerns.
What it does
Lets a client skip inline base64 when reading a session and fetch attachment bytes only when it needs them. It's experimental and opt-in at both ends: the client has to ask for refs, and the session's location needs
experimental.media_blobon, which is off by default. Every other client and session gets inline base64 exactly as today.Replaces #58657, which GitHub marked merged into its stack base while the stack was being reordered. Nothing from it reached
dev.Stack, built on the merged #58656: write (#58741, merged) → wire (this PR) → app (#59104) → backfill (#58742, optional, can land later).
Why
With media stored once in
blob(#58741), message reads still turn every ref back into base64, so each load of a long session with screenshots re-sends every image, a third larger than the raw bytes. With refs, a read carries a shortmedia:<hash>per file, and a client fetches each image once and caches it. Before → after bytes per read have not been measured yet.How it works
X-OpenCode-Capabilities: experimental-media-uri. Onlysession.messages,session.messageandsession.inbox.listhave ref variants. They return stored files asmedia:<hash>refs with their size inbytes. No route path is sent or stored.GET /api/experimental/media/:hash(generated client:experimental.media.get, OpenAPI tagexperimental) serves a blob only while a message or inbox item still references it. A blob that's been released or is waiting to be swept returns 404. Responses carry immutableCache-Control, anETagwith 304 support,nosniffand a sandbox CSP. Anything other than PNG, JPEG, GIF or WebP downloads asapplication/octet-stream.[media unavailable: <name> (<mime>)]placeholder instead of a dead link.media:<hash>ref back as an attachment. The server re-admits the stored bytes unchanged, without normalizing them again, and rejects a ref nothing references anymore.Risk
Server, protocol and the generated client. The new route and response shapes are experimental, and clients that don't opt in see no change. These still send media inline on purpose:
session.context, thesession.promptresponse, session export and import, and the runner's model requests. Legacy/api/media/URLs are not supported, since refs are the only form sent or accepted.Proof
bun test test/media.test.ts(packages/server): 2 pass. They cover the ref and inline shape switch; headers, ETag and 304; the missing-blob placeholder; following the flag from the session's project config when the server's own directory has it off; and a 404 plus a rejected prompt for an unreferenced blob.bun testfor the core prompt, owned, projector and store tests: 92 pass.bun run typecheck(core, server, client): clean.Added by the opencode bot (AI-generated).
Related: #36444, #58436
Public API: adds experimental GET /api/experimental/media/:hash (operationId experimental.media.get), an opt-in X-OpenCode-Capabilities: experimental-media-uri header for message/inbox reads, media: prompt attachments and new Media* schemas (backwards compatible)