Skip to content

Read the stream in blob(), formData() and macros when a Blob over its file cannot replace it - #48259

Draft
robobun wants to merge 3 commits into
mainfrom
robobun/b2ff6bbf/blob-from-one-shot-file-stream
Draft

robobun wants to merge 3 commits into
mainfrom
robobun/b2ff6bbf/blob-from-one-shot-file-stream

Conversation

@robobun

@robobun robobun commented Nov 8, 2026 •

Copy link
Copy Markdown
Collaborator

Problem

  • await new Response(Bun.stdin.stream()).blob() resolves with a Blob over fd 0. From a pipe, size is Infinity and text() gives abcde, then "". Bun.file(fd).stream() and a FIFO by path do the same. 1.3.14 held the bytes.
  • formData() over a Bun.file().stream() body resolves with no fields, even for a missing file. A macro that returns one inlines "".
  • Cause: the Source::File arm of ReadableStream::to_any_blob (src/runtime/webcore/ReadableStream.rs:214) gives every caller a Blob over the file of an unread stream.

Fix

  • to_any_blob takes what the caller does with the Blob (Lift). The File arm asks blob::file_store_lifts.
  • Keep (blob(), clone()) refuses a descriptor, a FIFO and a socket. InMemory (formData(), macros, fetch upload) refuses every file. Read, all other callers, is unchanged.
  • A reader whose stream cannot start rejects its promise.
  • Verified: test/js/web/fetch/body.test.ts (31 new cases, 20 fail on main), macro-test.test.ts (1). Suites: Notes.

Background

Downsides

Notes

Draft, with three questions for a maintainer. No issue reports this. A fuzz run found it.

  1. Order with http: consolidate the open fetch, Bun.serve, node:http, node:http2 and HTTP/3 pull requests #46528. That branch replaces the six reader prologues with one read_body() and a lift_unread_stream() with no argument (Body.rs:1701 at edb9ce8), and by reading it, text() lifts there too. On that branch this PR is the File arm plus one match action: GetBlob is Keep, GetFormData is InMemory. Which side carries it?
  2. A regular file by path. await new Response(Bun.file(path).stream()).blob() stays the lazy, typed Blob of the file, as on main and as clone() keeps it (body-clone.test.ts:1068). So the file can change under it, a missing path resolves and fails at the first read, and new Blob([b, b]) is 0 bytes (Blob constructor ignores File parts if more than one part is given #25851). A temporary file that the program removes after the await loses its bytes: blob.text() rejects with ENOENT. This is so since 1.4.0, and it was so in 1.2.23. 1.3.0 to 1.3.14 and the Fetch specification give a copy of the bytes with type "". Switching is one word: Lift::Keep to Lift::InMemory at BodyMixin::blob.
  3. A descriptor for the single-read callers. bytes(), arrayBuffer(), json(), Bun.serve and Bun.write still lift a descriptor. Over Bun.file(fd).stream() after readSync(fd, buf, 0, 3), bytes() gives lo txt for hello txt and moves the descriptor to the end. A Bun.serve handler that returns such a Response over one descriptor answers 1 of 3 requests. Lift::Read => !is_fd() fixes both. I built and measured it, and left it out: for a 64 MiB file bytes() goes from 33 to 67 ms and +67 MiB to 87 to 90 ms and +132 MiB, Bun.serve answers HEAD chunked, and these readers then also wait after process.stdin.pause(). It belongs with Read a regular-file stdin from its position in process.stdin and Bun.stdin.stream() #46700, Bun.stdin.stream(), process.stdin: read an unsliced file from its current offset #41687 and Bun.serve: read a Bun.file(fd) body at its offset, not at the descriptor's position #45266.

History (each release run with printf abcde | bun c.mjs):

release size, text(), text() again
1.1.26 5, abcde, abcde
1.1.27 to 1.2.23 Infinity, "", "" (#13704 added the lift to the readers)
1.3.0 to 1.3.14 5, abcde, abcde (#23313 moved the stream out of Locked.readable, so the lift found none)
1.4.0 to main Infinity, abcde, "" (#36624 keeps Locked.readable as a weak handle)

Rows that this PR changes, main 2596ac5 against this branch (release builds, Linux x64). stdin holds abcde:

main, pipe / < in.bin this branch
size, text(), text() again Infinity, abcde, "" / 5, abcde, "" 5, abcde, abcde
arrayBuffer() twice 5, 0 5, 5
slice(1, 3).text(), then text() ab, cde / bc, de bc, abcde
new Response(b).text() twice abcde, "" abcde, abcde
new Blob([b, b]) 0 bytes 10 bytes
Bun.write("out.bin", b) throws / 5 bytes 5 bytes
b.type application/octet-stream ""
the same through Request as above as above
formData() over a file stream, urlencoded [] the fields
formData() over a file stream, multipart rejects the fields
macro returns new Response(Bun.file(p).stream()) "" data:;base64,… as 1.3.14
r = new Response(Bun.file(fd)); r.body; await r.blob() lazy Blob holds the bytes
six readers over a stream of a closed or directory descriptor text() throws at the call, blob() and formData() resolve all reject (EBADF, EISDIR)
text() over Bun.file("/missing").stream() throws at the call rejects ENOENT
formData() over Bun.file("/missing").stream() resolves with no field rejects ENOENT

The last two rows are since 1.4.0. Each ran in five forms on 1.3.14, 1.4.4 and this branch: .catch() in an async function, the second argument of .then(), Promise.all, a try around the call, and a try around the await. Main 2596ac5 ran the .catch() form and matches 1.4.4.

  • text() of a Response or a Request: 1.2.23 and 1.3.14 reject. Since 1.4.0 the call throws, so no handler on its promise runs, in an async function too. Only a try sees the error. This branch rejects.
  • formData(): 1.3.14 throws at the call. 1.2.23 and 1.4.0 to main resolve with no field, so nothing sees an error. This branch rejects.
  • json() rejects on each release. arrayBuffer() and bytes() threw at the call on 1.3.14 and reject since 1.4.0.
  • blob() threw at the call on 1.3.14. 1.2.23, 1.4.0 to main and this branch give the Blob of the path (question 2).

Unchanged: new Response(Bun.stdin).blob() and Bun.stdin itself (the Blob the program gave), bytes() / arrayBuffer() / json(), Bun.serve, Bun.write, Bun.spawn stdin, shell redirects, socket writes, static routes.

Measurements. Release builds of main 2596ac5 and this branch. Tools: objdump, nm, size, a ptrace syscall counter, gdb single-step. perf, valgrind and strace are not installed here.

  • Non-stream bodies: Value::to_blob_if_possible runs the same 18 instructions and 2 conditional branches as on main.
  • stat on the JS thread for blob() over a path-file stream: +1 newfstatat per 1000 calls on one Bun.file, +1000 per 1000 fresh Bun.file, +0 when the file was already stat'd. bytes(), json(), arrayBuffer(): +0. On Windows the stat is skipped: by path it cannot tell a named pipe.
  • formData() over a path-file stream: main makes no syscall (and parses nothing). This branch: openat, fstat, 2 pread64, close per call, the same as text().
  • blob() over Bun.file(fd).stream(), 9 bytes: main reads nothing at the call, then three reads of the Blob give 9, 0, 0 bytes and the descriptor ends at 9. This branch reads at the call (2 fcntl, 1 fstat, 2 pread64, 1 close), the three reads give 9, 9, 9 with no syscall, and the descriptor stays at 0.
  • 64 MiB through blob() and one arrayBuffer(): main 44 ms and +65 MiB (and a second read is empty), this branch 184 ms and +197 MiB, 1.3.14 206 ms and +200 MiB. Bun.readableStreamToBlob(stream) is the same read on both builds.
  • PendingValue::set_promise for one text() of a JS stream body: 5,543 instructions on main, 5,609 here (+66, no allocation).
  • Release .text: +1,024 bytes. The stripped binary has the same size.
  • JS cells per call for the stream read: not measured. heapStats().objectCount only moves at a collection.

Left as they are, with their owners.

Suites on the debug build of this branch: body.test.ts 806 pass, body-clone.test.ts 100, body-stream.test.ts 9086, body-mixin-errors 23, body-async-iterator 50, body-stream-excess 4, macro-test.test.ts 35, FormData.test.ts 154, streams.test.js 727, html-rewriter.test.js 202, wasm-streaming 33, blob.test.ts 161, response 23, request 19, bun-serve-file 176, bun-serve-static 47, spawn-stdin-readable-stream 52, process-stdin 23, bunshell 464, readable-stream-blob-consumed, bun-stdin-slice, regression 29267. fetch.stream.test.ts has 18 timeouts, a subset of the 20 on a debug build of main. serve.test.ts: the privileged-port case fails as root, and one 10 s case sits at its limit on both builds. bun-write.test.js and spawn.test.ts have debug timeouts that differ per run, on main too. cargo check for x86_64-pc-windows-msvc and aarch64-apple-darwin, and cargo clippy -p bun_runtime, are clean. cargo mordant is not installed here.

Self-reviewed: 45 concerns raised. The review asked for this smaller cut in place of a required purpose at 28 call sites, and for the promise wrap, the kernel-pipe and multipart cases and the Windows stat gate: done. Not done: carrying the stdin fix of #48174 (named above), and a flag for whose stream it is (the .body row of the table is pinned as it stands).


no test proof · iteration 2 · platform-specific test(s) that do not run on this machine, deferring to CI, which covers all platforms: test/js/web/fetch/body.test.ts

…and in as a Blob

A Blob over a file holds no bytes: each read of it reads the file again.
ReadableStream::to_any_blob lifts the file out of an unread stream for
every caller. It now takes what the caller does with the Blob, and its
File arm decides:

- Read (unchanged): reads the Blob once, or hands the file to a child.
- Keep (blob(), clone()): not a descriptor, a FIFO or a socket. Those give
  their bytes once, so blob() reads the stream and holds the bytes.
- InMemory (formData(), macros, fetch upload): no file. These use the
  bytes where they are, and a Blob over a file has none.

A Body reader whose stream cannot start now rejects its promise. Before,
the exception was thrown at the call.
@robobun

robobun commented Nov 8, 2026 •

Copy link
Copy Markdown
Collaborator Author
Updated 8:06 PM PT - Nov 8th, 2026

❌ @robobun, your commit bab7745 has some failures in Build #138968 (All Failures)


🧪   To try this PR locally:

bunx bun-pr 48259

That installs a local version of the PR into your bun-48259 executable, so you can run:

bun-48259 --bun

@robobun

robobun commented Nov 8, 2026 •

Copy link
Copy Markdown
Collaborator Author

Status: draft. It waits for a maintainer's answer to the three questions in the Notes of the description.

How it was reproduced (Linux x64):

// c.mjs        printf abcde | bun c.mjs
const b = await new Response(Bun.stdin.stream()).blob();
console.log(b.size, await b.text(), JSON.stringify(await b.text()));
build output
1.1.26, 1.3.0, 1.3.14 5 abcde "abcde"
1.4.0, 1.4.4, main 2596ac5 Infinity abcde ""
this branch 5 abcde "abcde"

With bun c.mjs < in.bin main prints 5 abcde "".

// m.mjs        the file does not exist
const form = { headers: { "content-type": "application/x-www-form-urlencoded" } };
const r = new Response(Bun.file("zz-none.txt").stream(), form);
console.log(await r.formData().then(f => "fields: " + [...f].length, e => "rejects " + e.code));
build output
1.3.14 throws ENOENT at the call
1.2.23, 1.4.0, 1.4.4, main 2596ac5 fields: 0
this branch rejects ENOENT

On a debug build of main 2596ac5, 20 of the 31 new cases in test/js/web/fetch/body.test.ts fail. On a release build of main, the new case in test/bundler/transpiler/macro-test.test.ts fails. All pass on this branch.

CI (build 138968, bab7745b): the 184 jobs that ran passed. They cover Linux (glibc, musl, ASAN), Windows x64 and aarch64, and macOS aarch64. The build is marked failed for one reason: the four darwin x64 test jobs expired in the agent queue four times and did not run. The agents of that lane have a backlog. Five test files failed one time and passed on a retry or alone. This diff does not touch them.

Since 1.4.0, formData() resolves with no field over such a stream and
text() throws at the call. Both reject with ENOENT here. blob() is not in
the list: it gives the Blob of the path.
Comment thread src/runtime/webcore/Blob.rs Resolved
Comment thread src/runtime/webcore/Blob.rs Resolved
Comment thread src/runtime/webcore/Blob.rs Resolved
Comment thread src/runtime/webcore/Body.rs Resolved
Comment thread src/runtime/webcore/ReadableStream.rs Resolved
Comment thread src/runtime/webcore/ReadableStream.rs Resolved
Comment thread src/runtime/webcore/ReadableStream.rs Resolved

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants