Skip to content

feat(core): add byte-preserving offline media_blob backfill for legacy rows - #58742

Draft
kitlangton wants to merge 8 commits into
media-appfrom
media-backfill
Draft

kitlangton wants to merge 8 commits into
media-appfrom
media-backfill

Conversation

@kitlangton

@kitlangton kitlangton commented Oct 24, 2026 •

Copy link
Copy Markdown
Contributor

What it does

Adds an offline script that moves inline base64 media in existing session rows into the blob table, so databases written before the blob write path shrink too. It's experimental and something you run by hand. Nothing runs it automatically, and the experimental.media_blob flag (off by default) still decides only how new writes are stored.

Stack, built on the merged #58656: write (#58741, merged) → wire (#59103) → app (#59104) → backfill (this PR).

Optional offline cleanup: last in the stack, and it can land later. Clients don't need it.

bun packages/core/script/backfill-blobs.ts <path/to/opencode.db> [--vacuum-into <out.db>]

Why

Every image or file a user attached, and every tool screenshot, is stored as base64 inside its message row, a third larger than the raw bytes. The same image is copied again into the inbox row and into each update of a tool result. #58741 stops that for new writes. This handles the rows that already exist. Before → after database size has not been measured yet.

How it works

  1. It refuses to run unless the blob migration is recorded in the database, so start OpenCode once on it first. It never creates tables.
  2. For each session_message and session_inbox row that has inline media, it stores the decoded bytes once in blob, swaps in a media:<hash> ref and records a blob_ref row, using SessionBlob.split from the wire PR. It's byte-preserving: nothing is resized or re-encoded, so reads return the same data as before.
  3. Rows are decoded and hashed outside the lock, then written in small batches under BEGIN IMMEDIATE with a 1s busy timeout, so it fails fast when a server holds the lock. A row that changed after it was read is skipped. Re-running is safe: rows that already hold refs only get their missing blob_ref rows.
  4. --vacuum-into <out.db> writes a compacted copy. The database path is read correctly whether it comes before or after the flag.

Risk

It's an offline tool that rewrites rows in the database you point it at, so back it up first. Running it against a database a server is writing to fails fast instead of waiting. --vacuum-into needs free disk space about the size of the database. Reads already understand ref rows (from #58656), so backfilled rows read the same whether the flag is on or off.

Proof

  • bun test test/session-blob-backfill.test.ts (packages/core): 3 pass. They cover byte-for-byte storage, recorded refs and an idempotent re-run; refusal before the migration with the database untouched; skipping a row that changed mid-batch; and failing fast when another writer holds the lock. They ran with the core prompt, owned, projector and store tests at this head (95 pass).
  • bun run typecheck (core): clean. oxlint on changed files: 0 errors.

…l-media-uri

Clients that send X-OpenCode-Capabilities: experimental-media-uri get user
files and tool results stored in blob as media:<hash> refs with their size
and dimensions instead of inline base64, from session.messages,
session.message and session.inbox.list, when experimental.media_blob is on
in the session's own location (the config its media was written under).
Refs whose blob is gone become the same unavailable placeholder inline
readers get. Every other reader, the runner and export stay inline.

GET /api/experimental/media/:hash serves one blob that a stored message or
inbox item still references, with an immutable ETag, 304 support, nosniff
and a sandbox CSP; anything but a raster image is sent as an
application/octet-stream attachment. session.prompt accepts a referenced
media:<hash> ref as an attachment URI and re-admits its stored bytes as is.
The server sends media:<hash> refs, never route paths, and user files keep
their source, so drop the legacy route parsing and the origin fallback.
Reverting an undelivered prompt, undoing a queued prompt and editing one all end in an inbox cancel that sweeps blobs nothing else references. Each now copies its media: bytes through the media cache into local draft blobs first; revert and undo leave the prompt queued when the bytes cannot load.
…ssion rows

script/backfill-blobs.ts moves inline base64 attachments in existing session_message and session_inbox rows into blob and records their blob_ref rows, using SessionBlob.split so rows end up exactly as the live write path stores them. It refuses to run before the blob migration, batches under BEGIN IMMEDIATE with a short busy timeout, skips rows that change mid-run, is safe to re-run, and can VACUUM INTO a compacted copy.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant