feat(eve): send images to the model via toModelOutput content parts by ruiconti · Pull Request #1250 · vercel/eve

ruiconti · 2026-07-27T19:56:33Z

Implements #456 per the plan in #1245 (this PR is stacked on it — base is rui/tool-model-output-content-parts, review only the top commit).

A tool that produces an image can now hand the pixels to a vision-capable model instead of describing them. toModelOutput accepts { type: "content", value } with text and file parts, and new toolOutput.text / toolOutput.json / toolOutput.content and toolOutputPart.text / toolOutputPart.file builders from eve/tools hide the SDK's tagged data: { type: "data", data } nesting.

How it works

ToolModelOutput widens to mirror the AI SDK's ToolResultOutput content parts, narrowed to the JSON-safe subset: file data is the tagged FileData union restricted to { type: "data" } with a base64 string. The AI SDK bakes the projection into the persisted tool-result part once, so a base64 payload survives durable replay verbatim.
normalizeToolModelOutput — the single funnel for authored projections — rejects raw bytes (Uint8Array/ArrayBuffer/Buffer) with a base64 pointer even though the SDK accepts them: JSON.stringify corrupts them silently at the durable boundary, the same failure class as File parts fail the model call with image.source.base64.data: Input should be a valid string under @workflow/world-postgres #497. The url/reference/text tags are rejected as not yet supported, keeping the tag space open.
Payloads above 3 MiB warn instead of failing. Unlike sandbox-ref attachments (ref-only in history, hydrated per call), content parts persist in history and are re-sent on every subsequent model call — the docs section says so.
execute stays JSON-only and action.result keeps the raw output; channels never see the projection.

Capability epochs

The widened union reaches both authoring surfaces, so tool bumps to epoch 3 and dynamicTool to epoch 4, both retained (additive change). The --update tooling deadlocks when two capabilities mismatch at once — each run refuses while the other is dirty — so the bump is applied by hand exactly as the script writes it: version table, retained-epoch fixtures, regenerated report JSONs. guard:invariants passes. That deadlock probably deserves its own issue.

Testing

Unit coverage in tools.test.ts (happy path, builders, byte/tag/empty/unknown-part rejections, error identity). The e2e eval adds a render-pixel fixture tool whose execute returns only a label + base64 string — the pixels reach the model exclusively through the content parts, so the model naming the color proves the file part arrived as vision input.

Out of scope, tracked in the plan: sandbox-ref file data, URL/provider-reference tags, video/audio policy (#543), and the #497 user-message bug itself.

vercel · 2026-07-27T19:56:40Z

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
eve-docs	Ready	Preview, Comment, Open in v0	Jul 28, 2026 3:15pm
eve-docs-1644	Ready	Preview, Comment, Open in v0	Jul 28, 2026 3:15pm
eve-docs-4759	Ready	Preview, Comment, Open in v0	Jul 28, 2026 3:15pm

github-actions · 2026-07-27T19:59:24Z

Bundle + Package Summary: `apps/fixtures/weather-agent`

Key takeaways

No notable deltas vs rui/tool-model-output-content-parts (eb4ffa0).

Delta vs `rui/tool-model-output-content-parts (eb4ffa0)`

Area	Metric	Baseline	Current	Delta
Package	Packed tarball	7.42 MB	7.42 MB	+1.5 kB ⚠️
Package	Unpacked publish size	27.93 MB	27.94 MB	+8.6 kB ⚠️
Package	Installed footprint	68.82 MB	68.83 MB	+8.6 kB ⚠️
Package	Published files	2788	2792	+4
Package	Installed files	6257	6261	+4
Runtime	Unique function payloads	2	2	0
Runtime	Total function bytes	16.50 MB	16.50 MB	+4.1 kB ⚠️
Runtime	Public routes	11	11	0

Changed function payloads vs rui/tool-model-output-content-parts (eb4ffa0) (2)

Function	Status	Baseline	Current	Delta	Route changes
`functions/__server.func`	changed	8.25 MB	8.25 MB	+2.0 kB ⚠️	none
`functions/.well-known/workflow/v1/flow.func`	changed	8.25 MB	8.25 MB	+2.0 kB ⚠️	none

`eve init` install

Metric	Baseline	Current	Delta
Installed footprint	107.22 MB	107.23 MB	+8.6 kB ⚠️
Installed packages	123	123	0
dependencies	4	4	0
devDependencies	2	2	0
Dependency package bytes	42.30 MB	42.31 MB	+8.6 kB ⚠️
devDependency package bytes	5.04 MB	5.04 MB	0 B ➖

Build Metadata

Preset: vercel
Nitro: nitro@3.0.260610-beta
Output directory: apps/fixtures/weather-agent/.vercel/output
Build metadata timestamp: 2026-07-28T15:15:05.327Z
Route aliases: 11 public, 1 internal (12 total aliases)
Vercel routes in config: 14
Severity legend: 🔴 dominant/large, 🟠 notable, 🟡 watch, ⚪ small

Package Drill-Down

Package Details

Package: eve@0.27.8
Package directory: packages/eve
Tarball: 7.42 MB (eve-0.27.8.tgz)
Unpacked payload: 27.94 MB across 2792 published files
Installed footprint: 68.83 MB across 6261 installed files
Installed root package: 26.66 MB
Installed dependencies: 42.17 MB
Runtime dependencies: 2
Peer dependencies: 5 (4 optional)

Installed footprint is measured from an isolated temporary npm install of the packed tarball.

Heavy installed dependencies

eve: 26.66 MB (38.7%)
@rolldown/binding-linux-x64-gnu: 18.96 MB (27.5%)
ai: 6.51 MB (9.5%)
zod: 5.04 MB (7.3%)
nitro: 2.41 MB (3.5%)

Publish payload breakdown

Published file size
🔴 dist/src/compiled/shadcn-registry/index.js       [########################] 13.15 MB 47.1%
🟠 dist/src/compiled/experimental-ai-sdk-code-mo... [###.....................] 1.51 MB 5.4%
🟡 dist/src/compiled/@vercel/sandbox/index.js       [#.......................] 632.4 kB 2.3%
🟡 dist/src/compiled/_chunks/workflow/undici-C2Z... [#.......................] 502.4 kB 1.8%
🟡 dist/src/compiled/@chat-adapter/slack/index.js   [#.......................] 440.5 kB 1.6%
🔴 Other published files                            [#####################...] 11.71 MB 41.9%

Installed footprint breakdown

Installed package size
🔴 eve                             [########################] 26.66 MB 38.7%
🔴 @rolldown/binding-linux-x64-gnu [#################.......] 18.96 MB 27.5%
🔴 ai                              [######..................] 6.51 MB 9.5%
🔴 zod                             [#####...................] 5.04 MB 7.3%
🟠 nitro                           [##......................] 2.41 MB 3.5%
🟠 undici                          [#.......................] 1.62 MB 2.4%
🔴 Other installed packages        [#######.................] 7.63 MB 11.1%

Runtime dependencies (2)

Package	Range	Notes
`nitro`	`3.0.260610-beta`
`undici`	`7.28.0`

Peer dependencies (5)

Package	Range	Notes
`@opentelemetry/api`	`^1.0.0`	optional peer
`ai`	`catalog:`
`braintrust`	`^3.0.0`	optional peer
`just-bash`	`^3.0.0`	optional peer
`microsandbox`	`^0.5.0`	optional peer

eve init install drill-down

`eve init` install details

Command: eve init my-agent
Package manager: npm
Installed footprint: 107.23 MB across 8129 installed files
Installed packages: 123 total (117 transitive-only)
dependencies: 4 direct packages totaling 42.31 MB
devDependencies: 2 direct packages totaling 5.04 MB
Other transitive package files: 59.88 MB

Installed footprint is measured from an isolated temporary eve init my-agent using the current packed eve tarball.

Heavy installed dependencies

@typescript/typescript-linux-x64: 27.95 MB (26.1%)
eve: 26.66 MB (24.9%)
@rolldown/binding-linux-x64-gnu: 18.96 MB (17.7%)
zod: 9.00 MB (8.4%)
ai: 6.51 MB (6.1%)

Installed footprint breakdown

Installed package size
🔴 @typescript/typescript-linux-x64 [########################] 27.95 MB 26.1%
🔴 eve                              [#######################.] 26.66 MB 24.9%
🔴 @rolldown/binding-linux-x64-gnu  [################........] 18.96 MB 17.7%
🔴 zod                              [########................] 9.00 MB 8.4%
🔴 ai                               [######..................] 6.51 MB 6.1%
🟠 @types/node                      [##......................] 2.54 MB 2.4%
🔴 Other installed packages         [#############...........] 15.61 MB 14.6%

dependencies (4)

Package	Range	Installed size	Share
`@vercel/connect`	`0.4.2`	135.8 kB	0.1%
`ai`	`^7.0.38`	6.51 MB	6.1%
`eve`	`file:eve-0.27.8.tgz`	26.66 MB	24.9%
`zod`	`4.4.3`	9.00 MB	8.4%

devDependencies (2)

Package	Range	Installed size	Share
`@types/node`	`24.x`	2.54 MB	2.4%
`typescript`	`7.0.2`	2.50 MB	2.3%

Function Drill-Down

Payload Size Graph

Unique function payload size and share of total
🔴 functions/.well-known/workflow/v1/flow.func     [########################] 8.25 MB 50.0%
🔴 functions/__server.func                         [########################] 8.25 MB 50.0%

Top Function Payloads

🟠 functions/.well-known/workflow/v1/flow.func • 1 public route • 8.25 MB

Metric	Value
Public routes	`/.well-known/workflow/v1/flow`
Runtime	`nodejs24.x`
Handler	`index.mjs`
Payload	8.25 MB
Function files	8.25 MB across 44 files
Traced dependencies	0 B
Signal	🟠 Bundled file `_chunks/runtime-artifacts.mjs` is 1.58 MB (19.1%)

🟠 🔎 Dependency Analysis

📦 Bundled files:

Bundled file size
🟠 _chunks/runtime-artifacts.mjs [#############...........] 1.58 MB 19.1%
🟠 index.mjs                     [##########..............] 1.17 MB 14.2%
🟡 _libs/undici.mjs              [########................] 925.6 kB 11.2%
🟡 _chunks/world-vercel.mjs      [#######.................] 900.3 kB 10.9%
🟡 _chunks/sandbox.mjs           [######..................] 767.6 kB 9.3%
🟠 Other bundled files           [########################] 2.91 MB 35.3%

🧾 Vercel Config

{
  "handler": "index.mjs",
  "launcherType": "Nodejs",
  "shouldAddHelpers": false,
  "supportsResponseStreaming": true,
  "runtime": "nodejs24.x",
  "maxDuration": "max",
  "experimentalTriggers": [
    {
      "type": "queue/v2beta",
      "topic": "__eve776561746865722d6167656e74_wkf_workflow_*",
      "consumer": "default",
      "retryAfterSeconds": 5,
      "initialDelaySeconds": 0
    }
  ],
  "environment": {
    "WORKFLOW_PRECONDITION_GUARD": "1"
  }
}

🟠 functions/__server.func • 10 public routes, 1 internal alias • 8.25 MB

Metric	Value
Public routes	`/` `/eve/v1/callback/[token]` `/eve/v1/connections/[name]/callback/[token]` `/eve/v1/health` `/eve/v1/info` `/eve/v1/session` `/eve/v1/session/[sessionId]` `/eve/v1/session/[sessionId]/cancel` `/eve/v1/session/[sessionId]/stream` `/eve/v1/session/reset`
Internal aliases	`/__server`
Runtime	`nodejs24.x`
Handler	`index.mjs`
Payload	8.25 MB
Function files	8.25 MB across 44 files
Traced dependencies	0 B
Signal	🟠 Bundled file `_chunks/runtime-artifacts.mjs` is 1.58 MB (19.1%)

🟠 🔎 Dependency Analysis

📦 Bundled files:

Bundled file size
🟠 _chunks/runtime-artifacts.mjs [#############...........] 1.58 MB 19.1%
🟠 index.mjs                     [##########..............] 1.17 MB 14.2%
🟡 _libs/undici.mjs              [########................] 925.6 kB 11.2%
🟡 _chunks/world-vercel.mjs      [#######.................] 900.3 kB 10.9%
🟡 _chunks/sandbox.mjs           [######..................] 767.6 kB 9.3%
🟠 Other bundled files           [########################] 2.91 MB 35.3%

🧾 Vercel Config

{
  "handler": "index.mjs",
  "launcherType": "Nodejs",
  "shouldAddHelpers": false,
  "supportsResponseStreaming": true,
  "runtime": "nodejs24.x"
}

Build Timing: `e2e/fixtures/agent-tools-sandbox`

This is an informational timing measurement inside eve build, from preflight through publication. Output-size measurement and profile writing are excluded.

Build mode: deployable Vercel build with sandbox template prewarm included.

Build pipeline: 2.75 s -> 2.09 s (-661.2 ms) vs rui/tool-model-output-content-parts (eb4ffa0).
Timing is informational: shared GitHub runners are too variable for a hard timing budget.

Detailed phase timings vs `rui/tool-model-output-content-parts (eb4ffa0)`

Phase	Baseline	Current	Delta
`extension.check`	1.2 ms	1.2 ms	0.0 ms
`project.resolve`	0.7 ms	0.7 ms	0.0 ms
`workspace.create`	0.8 ms	0.8 ms	0.0 ms
`host.prepare`	921.3 ms	198.0 ms	-723.3 ms
`vercel.service-prefix.resolve`	2.5 ms	2.1 ms	-0.4 ms
`nitro.create`	217.2 ms	224.2 ms	+7.0 ms
`sandbox.prewarm`	310.8 ms	303.9 ms	-6.9 ms
`nitro.cache.prepare`	0.2 ms	0.3 ms	+0.1 ms
`nitro.prepare`	0.8 ms	0.9 ms	+0.1 ms
`nitro.public-assets`	0.8 ms	0.8 ms	0.0 ms
`nitro.prerender`	0.5 ms	0.6 ms	+0.1 ms
`nitro.bundle`	1.29 s	1.35 s	+62.2 ms
`nitro.cache.write`	0.4 ms	0.4 ms	0.0 ms
`agent-summary.emit`	0.6 ms	0.5 ms	-0.1 ms
`nitro.close`	0.1 ms	0.1 ms	0.0 ms
`output.publish`	3.8 ms	4.0 ms	+0.2 ms
`workspace.remove`	2.5 ms	2.3 ms	-0.2 ms

… parts ToolModelOutput gains a content variant mirroring the AI SDK's ToolResultOutput content parts narrowed to the JSON-safe subset: file data is the tagged FileData union restricted to { type: "data" } with a base64 string, so persisted tool results survive the durable JSON boundary verbatim. normalizeToolModelOutput rejects raw bytes (Uint8Array/ArrayBuffer/Buffer) with a base64 pointer — JSON.stringify would corrupt them silently, the #497 failure class — and rejects the url/reference/text FileData tags as not yet supported. Payloads above the 3 MiB inline-image reference point warn: unlike sandbox-ref attachments, content parts persist in history and are re-sent on every subsequent model call. New toolOutputPart.text/file builders ship from eve/tools so authors skip the tagged nesting. execute stays JSON-only and action.result keeps the raw output. The tool and dynamicTool capability epochs bump (retained) since the widened union reaches both authoring surfaces. Implements the plan in research/tool-model-output-content-parts.md (#456). Signed-off-by: Rui Conti <ruiconti@gmail.com>

Companion to toolOutputPart: toolOutput.text/json/content build the model-facing ToolModelOutput union so authored projections compose as toolOutput.content([toolOutputPart.text(...), toolOutputPart.file(...)]). Pure sugar — hand-written literals remain valid. The module renames to output-builders.ts since it now holds output builders, not just parts. The tool capability epoch-3 report is regenerated in place rather than bumped: epoch 3 is introduced by this branch and unreleased, so its hash is not yet published contract history. Signed-off-by: Rui Conti <ruiconti@gmail.com>

… its own module tools.ts is the tool-set assembly module; the content-parts change had grown a second concern inside it — a ~165-line validation subsystem for authored toModelOutput results. It moves to tool-model-output.ts next to the serialization error machinery it throws. Exporting the normalizer also deletes the test ceremony: the shape matrix no longer builds a HarnessToolMap and fishes the toModelOutput closure out of the SDK tool with casts per case — it calls the function. tools.test.ts keeps one wiring test proving buildToolSet applies the normalizer. Signed-off-by: Rui Conti <ruiconti@gmail.com>

The hand-rolled probe-cast guards in the normalizer are replaced by a zod discriminated union (vendored #compiled/zod + formatValidationError, the established untrusted-boundary pattern). Structural rejections now carry path-labeled messages like: Invalid discriminator value. Expected 'text' | 'file' at "value[0].type" Two author mistakes stay hand-written pre-checks because the schema cannot explain them: raw bytes (zod would report a missing data.type key on a Uint8Array; the real fix is base64-encoding — and silent JSON.stringify corruption is the #497 failure class) and the not-yet- supported url/reference/text FileData tags. As a side effect the byte rejection is now checked once for both payload positions. Signed-off-by: Rui Conti <ruiconti@gmail.com>

The hand-written 1x1 base64 was corrupt; OpenAI and Anthropic both rejected it at the model call ('image data does not represent a valid image'). Replaced with a programmatically generated 64x64 solid-red PNG (132 bytes) verified to decode — 64x64 also clears any provider minimum-size rules. The unit-test constant is swapped to the same valid payload so the corrupt string cannot be copy-pasted from the repo. Signed-off-by: Rui Conti <ruiconti@gmail.com>

The discover and build-extension tests hardcoded tool contract v2/v3 literals, so the epoch bump in this branch aged them out (and any future bump would again). The unsupported-version fixture now requires current + 1 — unsupported by construction — and the stamped-manifest assertion tracks EXTENSION_CAPABILITY_VERSIONS; the real assertions (rejection diagnostics, the set of stamped capabilities) are unchanged. Signed-off-by: Rui Conti <ruiconti@gmail.com>

assertSupportedFilePartData says what holds after the call; the old rejectTeachableFilePartMistakes described the error strategy instead. Signed-off-by: Rui Conti <ruiconti@gmail.com>

The render-pixel eval asserted a fixed answer ('red') — the one color a blind model guesses most, so a silently dropped file part could still pass. render-stripes replaces it: three colors are shuffled per run from a palette of unambiguous single-token names, rendered as vertical stripes, and recorded on action.result as the answer key. eventsSatisfy joins that key with the assistant reply and requires the colors named in left-to-right order — spatial evidence a blind guess cannot fake, while the eval stays self-contained. A second turn asks for the colors again without re-running the tool (calledTool count stays 1), witnessing that the content part persisted in history re-renders on replay. The output matcher now also pins the raw-execute-output side of the contract: action.result carries the answer key and the base64 payload the model never sees as JSON. Signed-off-by: Rui Conti <ruiconti@gmail.com>

ruiconti force-pushed the rui/tool-model-output-content-parts-impl branch from 9f5e9d1 to a25d40b Compare July 27, 2026 20:04

vercel Bot deployed to Preview – eve-docs-4759 July 27, 2026 20:05 View deployment

vercel Bot deployed to Preview – eve-docs July 27, 2026 20:06 View deployment

vercel Bot deployed to Preview – eve-docs-1644 July 27, 2026 20:06 View deployment

AndrewBarba approved these changes Jul 27, 2026

View reviewed changes

ruiconti mentioned this pull request Jul 27, 2026

fix(eve): stub content-output file payloads in the compaction transcript #1255

Merged

vercel Bot deployed to Preview – eve-docs-4759 July 27, 2026 20:45 View deployment

vercel Bot deployed to Preview – eve-docs-1644 July 27, 2026 20:45 View deployment

vercel Bot deployed to Preview – eve-docs July 27, 2026 20:45 View deployment

vercel Bot deployed to Preview – eve-docs-4759 July 27, 2026 21:05 View deployment

vercel Bot deployed to Preview – eve-docs-1644 July 27, 2026 21:05 View deployment

vercel Bot deployed to Preview – eve-docs July 27, 2026 21:05 View deployment

vercel Bot deployed to Preview – eve-docs-4759 July 27, 2026 21:15 View deployment

vercel Bot deployed to Preview – eve-docs-1644 July 27, 2026 21:15 View deployment

vercel Bot deployed to Preview – eve-docs July 27, 2026 21:21 View deployment

ruiconti force-pushed the rui/tool-model-output-content-parts-impl branch from b9ec631 to c273037 Compare July 27, 2026 21:42

vercel Bot deployed to Preview – eve-docs-4759 July 27, 2026 21:44 View deployment

vercel Bot deployed to Preview – eve-docs-1644 July 27, 2026 21:44 View deployment

vercel Bot deployed to Preview – eve-docs July 27, 2026 21:45 View deployment

vercel Bot deployed to Preview – eve-docs-4759 July 27, 2026 21:55 View deployment

vercel Bot deployed to Preview – eve-docs-1644 July 27, 2026 21:56 View deployment

vercel Bot deployed to Preview – eve-docs July 27, 2026 21:56 View deployment

vercel Bot temporarily deployed to Preview – eve-docs-4759 July 28, 2026 03:58 Inactive

vercel Bot temporarily deployed to Preview – eve-docs-1644 July 28, 2026 03:58 Inactive

vercel Bot deployed to Preview – eve-docs July 28, 2026 03:59 View deployment

ruiconti added 7 commits July 28, 2026 11:12

refactor(eve): harness - name the file-part pre-check as an assertion

caad897

assertSupportedFilePartData says what holds after the call; the old rejectTeachableFilePartMistakes described the error strategy instead. Signed-off-by: Rui Conti <ruiconti@gmail.com>

ruiconti force-pushed the rui/tool-model-output-content-parts-impl branch from e0dbf63 to 862bd05 Compare July 28, 2026 15:12

vercel Bot deployed to Preview – eve-docs-1644 July 28, 2026 15:13 View deployment

vercel Bot deployed to Preview – eve-docs-4759 July 28, 2026 15:13 View deployment

vercel Bot deployed to Preview – eve-docs July 28, 2026 15:15 View deployment

ruiconti merged commit c804141 into main Jul 28, 2026
106 of 107 checks passed

ruiconti deleted the rui/tool-model-output-content-parts-impl branch July 28, 2026 16:35

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat(eve): send images to the model via toModelOutput content parts - #1250

feat(eve): send images to the model via toModelOutput content parts#1250
ruiconti merged 8 commits into
rui/tool-model-output-content-partsfrom
rui/tool-model-output-content-parts-impl

ruiconti commented Jul 27, 2026 •

edited

Loading

Uh oh!

vercel Bot commented Jul 27, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Jul 27, 2026 •

edited

Loading

Package Details

`eve init` install details

Payload Size Graph

Top Function Payloads

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

ruiconti commented Jul 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

vercel Bot commented Jul 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jul 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Bundle + Package Summary: apps/fixtures/weather-agent

Delta vs rui/tool-model-output-content-parts (eb4ffa0)

eve init install

Package Details

eve init install details

Payload Size Graph

Top Function Payloads

Build Timing: e2e/fixtures/agent-tools-sandbox

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

ruiconti commented Jul 27, 2026 •

edited

Loading

vercel Bot commented Jul 27, 2026 •

edited

Loading

github-actions Bot commented Jul 27, 2026 •

edited

Loading

Bundle + Package Summary: `apps/fixtures/weather-agent`

Delta vs `rui/tool-model-output-content-parts (eb4ffa0)`

`eve init` install

`eve init` install details

Build Timing: `e2e/fixtures/agent-tools-sandbox`