Skip to content

udp: fall back to recvmsg/sendmsg when a seccomp filter refuses recvmmsg/sendmmsg - #42678

Open
robobun wants to merge 2 commits into
mainfrom
robobun/5d15e596/udp-mmsg-runtime-fallback
Open

robobun wants to merge 2 commits into
mainfrom
robobun/5d15e596/udp-mmsg-runtime-fallback

Conversation

@robobun

@robobun robobun commented Sep 14, 2026 •

Copy link
Copy Markdown
Collaborator

Problem

  • A seccomp filter can refuse recvmmsg(2) / sendmmsg(2) with ENOSYS or EPERM and still allow recvmsg(2) / sendmsg(2). With recvmmsg refused, no datagram arrives, 'error' fires about 250,000 times per second (server 'error': ENOSYS), and the event loop uses 100 % CPU.
  • With sendmmsg refused, every socket.send fails with that errno, and an HTTP/3 fetch spins until it times out. Node works under the same filter.
  • The cause: bsd_recvmmsg / bsd_sendmmsg (packages/bun-usockets/src/bsd.c:147-195) and us_quic_packets_out (quic.c:283) have no run-time fallback.

Fix

  • bsd_recvmmsg and the new bsd_sendmmsg_msgvec retry an ENOSYS / EPERM failure with one recvmsg / sendmsg per datagram. bsd_sendmmsg and us_quic_packets_out both call it. The loops keep the return contract of the batched calls (the count moved, or -1 when the first fails), so loop.c and udp.c do not change.
  • A process-wide flag then skips the batched syscall. It is set only when the per-datagram syscall answers differently, because netfilter also rejects a datagram with EPERM.
  • Verified: test/js/bun/udp/udp-mmsg-unavailable-linux.test.ts (9 cases, the released build fails all). Also test/js/bun/udp/, two HTTP/3 fetch suites and Node's test-dgram-*.js, with and without the filter.
  • Self-reviewed: the surviving concerns were about wording. The comments no longer say that WSL1 or qemu-user lack these syscalls.

Background

  • recvmmsg / sendmmsg move several datagrams in one syscall. Bun uses them for every datagram of node:dgram, Bun.udpSocket and QUIC.
  • libuv receives with recvmsg and sends one datagram with sendmsg, so Node does not need the batched calls.
  • uSockets polls level-triggered. A failed receive leaves the datagram queued, so the socket is readable again at once.
Notes

Reach. No user report exists for this. Docker's default profile, systemd's @network-io group, gVisor, WSL1 and qemu-user all allow both syscalls today. The fix reaches hand-written seccomp filters (an allowlist or a denylist that leaves out recvmmsg / sendmmsg). The repro in the report is systemd-run -p SystemCallFilter=~recvmmsg -p SystemCallErrorNumber=ENOSYS.

Repro without systemd. A small launcher installs a seccomp filter that fails the two syscalls with a given errno, then execs the runtime (the test compiles the same launcher with cc, like test/js/node/fs/fs-stat-seccomp-linux.test.ts). A node:dgram echo script for 3 s, released bun 1.4.3-canary (b993710):

filter result
none echo: ping, 1 CPU ms
recvmmsg → ENOSYS NO MESSAGE in 3 s, 941,695 'error' events, 4273 CPU ms
recvmmsg → EPERM NO MESSAGE in 3 s, 1,050,216 'error' events, 4284 CPU ms
sendmmsg → ENOSYS / EPERM send cb: ENOSYS / EPERM, no echo
both, HTTP/3 fetch The operation timed out, 4134 CPU ms
both, Node v26.3.0 echo: ping, 4 CPU ms

With this change every row prints echo: ping (or ok for HTTP/3) with 0 'error' events.

The flag. An LD_PRELOAD shim that logs the four libc calls shows one sendmmsg and one recvmmsg attempt per process, then only sendmsg / recvmsg. With sendmmsg and sendmsg both failing with EPERM (a stand-in for a netfilter OUTPUT DROP), the flag stays clear, the user still gets EPERM, and the next send tries sendmmsg again.

Mid-batch errors. When a recvmsg after the first one fails, the loop returns the count so far and the errno is dropped. The kernel's recvmmsg tries to keep it in sk_err for the next call (see BUGS in recvmmsg(2)). Sockets that Bun creates with an error handler set IP_RECVERR, so the error is still in the error queue and loop.c reports it from there.

Suites run with the debug build. With and without the filter (both syscalls refused, ENOSYS and EPERM): test/js/bun/udp/udp_socket.test.ts (218), dgram.test.ts (62), udp_socket_recv_flags.test.ts (2). With and without the filter (ENOSYS): test/js/web/fetch/fetch-http3-client.test.ts (56), fetch-http3-syscall-fault.test.ts (3), and the 75 test/js/node/test/parallel/test-dgram-*.js files. The released build fails 32 of those 75 under the filter. test/js/node/dgram/node-dgram.test.js has one failure in this container with and without the change (ENODEV from addMembership("ff02::1"), no IPv6 multicast interface).

Not in this change.


no test proof · iteration 2 · platform-specific test(s) that do not run on this machine, deferring to CI, which covers all platforms: test/js/bun/udp/udp-mmsg-unavailable-linux.test.ts

…msg/sendmmsg

A seccomp filter can answer recvmmsg(2) and sendmmsg(2) with ENOSYS or EPERM
and still allow recvmsg(2) and sendmsg(2). With recvmmsg refused no datagram
was delivered and the level-triggered readable poll spun the event loop.
With sendmmsg refused every UDP send failed, and the QUIC send path spun on
its writable poll.

bsd_recvmmsg and the new bsd_sendmmsg_msgvec (used by bsd_sendmmsg and by
us_quic_packets_out) now retry such a failure with one recvmsg(2) or
sendmsg(2) per datagram. When that syscall gives a different answer, a
process-wide flag skips the batched syscall from then on.
@robobun

robobun commented Sep 14, 2026 •

Copy link
Copy Markdown
Collaborator Author

Status: fix pushed in #42678. The diff is green. It needs a maintainer to review and merge.

How it was reproduced. A small launcher installs a seccomp filter that fails recvmmsg and/or sendmmsg with ENOSYS or EPERM, then execs the runtime. A node:dgram echo script ran under it with the released build (1.4.3-canary b993710):

  • recvmmsg refused: NO MESSAGE in 3 s, 941,695 'error' events (server 'error': ENOSYS), 4273 CPU ms in a 3 s wait.
  • sendmmsg refused: send cb: ENOSYS, no echo.
  • Both refused, HTTP/3 fetch: The operation timed out, 4134 CPU ms.
  • Node v26.3.0 under the same filter: echo: ping, 4 CPU ms.

The new test test/js/bun/udp/udp-mmsg-unavailable-linux.test.ts compiles the same launcher. The released build fails 9 of 9 cases. The build from this branch passes 9 of 9.

CI. Build #115390: 181 of 182 jobs passed, no test failed. The remaining job (darwin any aarch64 - test-bun, shard 0 of 2) timed out with no test output after its guest started. macOS does not compile this change (#if !defined(_WIN32) && !defined(__APPLE__)) and the new test is skipped there. All Linux test jobs passed in both builds. The job logs of #115383 show the new test at 9 pass, 0 fail on Debian 13 x64, Debian 13 x64-asan and Alpine 3.23 x64. The only failure in #115383 was test/cli/install/bun-patch.test.ts on Windows 2019 x64. It passed in #115390.

@coderabbitai

coderabbitai Bot commented Sep 14, 2026 •

Copy link
Copy Markdown
Contributor

Review Change StackReview Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Essentials

Run ID: 8ba4cf18-391b-4b7f-bc68-c56e3d6cff75

📥 Commits

Reviewing files that changed from the base of the PR and between 64689a3 and 3f2b707.

📒 Files selected for processing (4)
  • packages/bun-usockets/src/bsd.c
  • packages/bun-usockets/src/internal/networking/bsd.h
  • packages/bun-usockets/src/quic.c
  • test/js/bun/udp/udp-mmsg-unavailable-linux.test.ts

Included review availability: Your plan provides up to 10 included reviews per hour; 3 remain after this review.


Walkthrough

Changes

The Unix socket layer now provides cached per-message fallbacks for unavailable sendmmsg and recvmmsg calls. Linux QUIC sends use the send wrapper. Linux seccomp tests cover UDP and HTTP/3 behavior.

Batched socket syscall fallback

Layer / File(s) Summary
Compatibility wrappers
packages/bun-usockets/src/internal/networking/bsd.h, packages/bun-usockets/src/bsd.c
Adds the platform-gated send wrapper. Send and receive paths retry EINTR, preserve partial progress, and cache confirmed ENOSYS or eligible EPERM results before using per-message operations.
QUIC send integration
packages/bun-usockets/src/quic.c
Routes Linux QUIC batched sends and retries through bsd_sendmmsg_msgvec.
Seccomp regression coverage
test/js/bun/udp/udp-mmsg-unavailable-linux.test.ts
Builds a seccomp launcher and tests Node UDP echoes, Bun.udpSocket.sendMany(), and HTTP/3 when batched syscalls return ENOSYS or EPERM.

Suggested reviewers: cirospaciari

Priority: ➖ Normal

Merge Risk: ⚪ Minimal · up to 3f2b7

The fallback and its regression coverage have no established merge-blocking issue.

🚥 Pre-merge checks | ✅ 4
✅ Passed checks (4 passed)
Check name Status Explanation
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Title check ✅ Passed The title clearly and concisely describes the primary change: adding recvmsg/sendmsg fallbacks when seccomp filters reject recvmmsg/sendmmsg.
Description check ✅ Passed The description explains the problem, fix, behavior, scope, verification steps, test results, and known limitations. It does not use the exact template headings, but it provides the required informati…

Comment @coderabbitai help to get the list of available commands.

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I reviewed this and found no bugs. Since it changes the native UDP receive/send hot path in usockets and adds process-global syscall-availability latches, a human look is still worthwhile.

What was reviewed:

  • Fallback loops preserve the mmsg return contract (count-so-far or -1 on first failure, msg_len set); non-blocking recv returns the partial count on EAGAIN.
  • EPERM-ambiguity guard: flag latches only when the per-message result differs, so a netfilter EPERM on a single datagram doesn't disable batching.
  • Platform gating: bsd_sendmmsg_msgvec is declared/defined under !_WIN32 && !__APPLE__ and the quic.c caller is under __linux__; MSG_NOSIGNAL and EINTR retries are preserved at both replaced sites.
  • Test wires every socket error to done() so the pre-fix failure mode exits fast rather than hanging; uses port 0, test.concurrent, and skips cleanly when cc/seccomp are unavailable.
Extended reasoning...

Overview

This PR adds a runtime fallback in packages/bun-usockets/src/bsd.c from the Linux batched recvmmsg(2)/sendmmsg(2) syscalls to per-datagram recvmsg(2)/sendmsg(2) when a seccomp filter refuses the batched calls with ENOSYS or EPERM. It introduces two process-global relaxed-atomic latches, two static helper loops that emulate the mmsg return contract, and a new exported bsd_sendmmsg_msgvec used by both bsd_sendmmsg and the QUIC send path in quic.c. A new Linux-only test compiles a seccomp launcher and verifies node:dgram, Bun.udpSocket.sendMany, and HTTP/3 fetch all still work under six deny×errno combinations.

Security risks

None identified. The change is confined to syscall selection on the send/receive path and does not touch input parsing, TLS, auth, or any trust boundary. The seccomp launcher in the test is test-only C compiled at test time and never ships. The relaxed-atomic latches are monotonic booleans read/written only for fast-path selection; misordering cannot cause anything worse than one extra sendmmsg/recvmmsg attempt.

Level of scrutiny

Moderate-to-high. bsd_recvmmsg/bsd_sendmmsg sit on the hot path for every UDP datagram in node:dgram, Bun.udpSocket, and QUIC/HTTP3, across the JS thread and the HTTP thread. The change is small and mechanical in shape, but it introduces process-global state, alters syscall selection under an ambiguous errno (EPERM can come from either seccomp or netfilter), and replaces two direct sendmmsg sites in quic.c. The EPERM disambiguation ("only latch when the per-message result differs") is the load-bearing non-obvious choice; it looks correct — a netfilter drop yields the same EPERM from sendmsg, so the flag stays clear and the next send retries sendmmsg — but a maintainer who owns usockets should confirm they agree with that tradeoff and with a process-wide (not per-fd) latch.

Other factors

I verified: the header declaration guard matches the definition guard exactly and the only external caller (quic.c) is under #if defined(__linux__), so non-Linux targets never reference the symbol; EINTR retry and MSG_NOSIGNAL are preserved at every replaced site; the removed (struct mmsghdr *) cast on recvbuf->msgvec is safe because the field is already struct mmsghdr[LIBUS_UDP_RECV_COUNT]; the fallback loops set msg_len and return i > 0 ? i : -1, matching the kernel contract callers in loop.c/udp.c depend on. The test follows harness conventions (bunEnv, bunExe, port: 0, test.concurrent, tempDirWithFiles), wires every error event to resolve so the pre-fix spin manifests as an immediate failure line rather than a timeout, and mirrors the existing fs-stat-seccomp-linux.test.ts pattern for the C launcher. No CODEOWNERS entry covers the changed paths. Given the hot-path native networking surface, deferring to a human is the right call even with a clean automated pass.

@robobun

robobun commented Sep 14, 2026 •

Copy link
Copy Markdown
Collaborator Author
Updated 11:18 PM PT - Sep 13th, 2026

❌ @robobun, your commit 4e1d920 has some failures in Build #115390 (All Failures)


🧪   To try this PR locally:

bunx bun-pr 42678

That installs a local version of the PR into your bun-42678 executable, so you can run:

bun-42678 --bun

Jarred-Sumner pushed a commit that referenced this pull request Sep 14, 2026
…he HTTP/3 engine (#42732)

### Problem
- HTTP/3 uses more than 100% CPU when UDP sends keep failing with an
errno other than `EAGAIN`. Both `fetch()` and `Bun.serve({ http3: true
})`.
- Two triggers: an iptables DROP rule (`EPERM`), and an egress MTU below
1500, where each DPLPMTUD probe fails with `EMSGSIZE`.
- The cause is the tail of `us_quic_packets_out`
(`packages/bun-usockets/src/quic.c:316`). It reports every send errno as
`EAGAIN` and arms `LIBUS_SOCKET_WRITABLE`. The socket is already
writable, so `on_drain` fires at once and the send fails again.

### Fix
- `us_quic_packets_out` passes `EMSGSIZE` through. lsquic retires that
datagram with `ci_packet_too_large` and feeds DPLPMTUD.
- Any other refused datagram counts as sent. To lsquic that is a packet
lost on the wire, so its loss timer paces the retries and a timeout ends
the connection.
- Only `EAGAIN`, `EWOULDBLOCK` and `ENOBUFS` still return short and arm
`WRITABLE`. quinn draws the same line.
- Verified: `fetch-http3-syscall-fault.test.ts`, four new tests, three
fail on stock bun. Also the other h3 suites and `node:quic`, on Linux
and Windows.

### Background
- lsquic calls `us_quic_packets_out` to send a batch of datagrams. One
engine serves every HTTP/3 connection of the process, so one refused
peer stalls the rest.
- A short return with `EAGAIN` stops every send of that engine until
`lsquic_engine_send_unsent_packets` runs. Bun runs it from `on_drain`,
the callback of a writable UDP poll.
- A short return with `EMSGSIZE` retires one datagram and leaves the
engine running.
- Any other errno closes the connection that owns the first unsent
datagram. A pending ICMP error on the shared client socket belongs to no
particular peer, so that does not fit.

<details><summary>Notes</summary>

**Reach: what was run, and what was simulated.** The
unsendable-IP-literal trigger is real (`EACCES` from the kernel, no
privileges needed). `EPERM` from a firewall rule is simulated two ways:
a seccomp filter that fails `sendmsg`/`sendmmsg`, and
`US_FAULT_SENDMSG`. The report this came from also saw it with a real
`iptables -A OUTPUT -p udp --dport <port> -j DROP`. The low egress MTU
is simulated with an `LD_PRELOAD` shim, because the container has no
`CAP_NET_ADMIN` to set a link MTU. No user has reported this. The HTTP/3
API is experimental, so this is a low-priority fix.

**Repro without privileges.** `fetch("https://255.255.255.255/", {
protocol: "http3", signal: AbortSignal.timeout(1500) })`. An IP literal
skips DNS and the connect-time route probe
(`us_quic_socket_context_connect`), and `sendmsg` to the limited
broadcast address fails every time on a socket without `SO_BROADCAST`.
Unset `HTTPS_PROXY` first, a proxy makes fetch refuse HTTP/3 up front.
Stock bun uses 2056 ms of CPU in a 1502 ms wait, and 1354 ms of it after
the abort. With the fix: 69 ms, then 25 ms.

**Tests.**
- `a datagram the kernel refuses to send is dropped without spinning the
HTTP thread` uses that address in a child process and compares
`process.cpuUsage()` with the wall time of a 1 s wait. It needs no fault
injection, so it runs on release builds too. Unfixed: 1365 ms CPU
(release), 1033 ms (debug). Fixed: 34 ms. Skipped on Windows, where the
dual-stack socket sends to `::ffff:255.255.255.255` without an error.
- `Bun.serve: a send error on every datagram does not spin the server's
event loop` arms `EPERM` on every send from inside a request handler and
reads the server child's CPU time. Spin: 2124 ms CPU in 1050 ms. Fixed:
135 ms.
- `Bun.serve: a refused version-negotiation reply does not spin the loop
when no connection exists` sends one long-header datagram with version
`0x0a0a0a0a`, built from a real Initial with its version bytes replaced.
The server queues a Version Negotiation reply for a connection it does
not have. The test first sends the datagram to an unfaulted server and
checks that the reply is a Version Negotiation packet that offers QUIC
v1, so the CPU check cannot pass without a reply to refuse. This is the
only spin with no connection and no timeout to end it. Spin: 2129 ms CPU
in 1052 ms. Fixed: 325 ms against a 150 ms idle baseline for a debug
ASAN server. On the release binary under seccomp the old code uses 2637
ms in 2002 ms.
- `a DPLPMTUD probe above the egress MTU does not stall the transfer`
streams 2.5 s of body under an `LD_PRELOAD` shim that refuses any
datagram above 1400 payload bytes, which is above lsquic's base packet
size and below its first probe. Unfixed: the transfer delivers 32 KB,
stalls for the remaining 11 s, and the next request on the same engine
times out. Fixed: 3.1 MB delivered, no stall, 696 ms CPU in 4049 ms, and
each probe refused once instead of twice.
- The MTU shim declares `sendmmsg` with the flags type of the libc it
builds against: `int` on glibc, `unsigned int` on musl.
- An injected 0 from `US_FAULT_SENDMSG` (the `zero` action) now counts
as one datagram sent in the `sendmmsg` path, as it already does in
`us_quic_send_one`. Before, it left that loop without progress. Only the
test hook can reach this.
- The two `Bun.serve` tests pass on stock bun for a different reason:
the old `sendmmsg` retry does not go through `US_FAULT_SENDMSG`, so an
injected error cannot persist and the retry sends the datagram. This PR
puts the retry behind the hook. The CPU numbers above come from a build
with the hook and the old errno logic.

**`IP_PMTUDISC_PROBE`.** The comment in `us_quic_set_dontfrag` claimed
that this option lets DPLPMTUD probe without `EMSGSIZE`. It does not. It
makes the kernel ignore its cached path MTU. The device MTU still bounds
the datagram: on a 1500 MTU link, a 1472-byte payload is sent and 1473
bytes gives `EMSGSIZE`. The comment now says that, and names where the
errno goes.

**Prior art.** quinn waits for a writable socket only on `WouldBlock`.
It logs every other send error and goes on (`UdpSenderHelper::poll_send`
in `quinn/src/runtime/mod.rs`, with the test
`non_would_block_send_errors_are_logged_and_ignored` using
`PermissionDenied`). Its `quinn-udp` doc says UDP send errors are
non-fatal "because higher-level protocols must employ retransmits and
timeouts anyway".

**Not in this change.** `node:quic`'s own `packets_out`
(`src/runtime/node/quic/endpoint.rs:792-803`) has the same errno
rewrite. It does not spin: `us_udp_socket_send` arms `WRITABLE` only for
`EAGAIN`, so each failure costs one second through lsquic's failsafe. It
also keeps `EMSGSIZE`, as this PR now does. A classifier change there
belongs in its own PR, because #40220 rewrites that file (+645/-1056).

**Relation to #42678.** Both change the same two `sendmmsg` calls, and
#42678 calls this behaviour "a separate bug". This PR is the smaller of
the two, so it should land first. The conflict is mechanical: the call
moves into `us_quic_sendmmsg`.

**Other cases that stay as they are.** `ENOBUFS` remains backpressure,
as #35255 chose. That holds on Windows too: `us_quic_send_one` keeps
`WSAENOBUFS` as `ENOBUFS`. Before, it became `EIO`, which the old tail
still treated as backpressure and this PR would have dropped. A UDP
socket that is already closed (`!ls->udp`) still returns short with no
poll to arm, so lsquic's one-second failsafe paces it.

**Self-reviewed:** 8 concerns raised, 5 addressed (the `EMSGSIZE`
classifier and its false comment, the zero-connection spin, the
`node:quic` twin note, the #42678 note, the reach note). 3 rejected:
folding the `node:quic` fix in (#40220 rewrites that file), patching
lsquic instead (the callback contract has no fourth outcome), and a
client-side address-walk change (a separate behaviour, not this bug).

</details>

<!-- robobun:evidence:begin -->

---

**no test proof** · iteration 0 · platform-specific test(s) that do not
run on this machine, deferring to CI, which covers all platforms:
test/js/web/fetch/fetch-http3-syscall-fault.test.ts

<!-- robobun:evidence:end -->
usrbinkat pushed a commit to usrbinkat/bun that referenced this pull request Sep 15, 2026
…he HTTP/3 engine (oven-sh#42732)

### Problem
- HTTP/3 uses more than 100% CPU when UDP sends keep failing with an
errno other than `EAGAIN`. Both `fetch()` and `Bun.serve({ http3: true
})`.
- Two triggers: an iptables DROP rule (`EPERM`), and an egress MTU below
1500, where each DPLPMTUD probe fails with `EMSGSIZE`.
- The cause is the tail of `us_quic_packets_out`
(`packages/bun-usockets/src/quic.c:316`). It reports every send errno as
`EAGAIN` and arms `LIBUS_SOCKET_WRITABLE`. The socket is already
writable, so `on_drain` fires at once and the send fails again.

### Fix
- `us_quic_packets_out` passes `EMSGSIZE` through. lsquic retires that
datagram with `ci_packet_too_large` and feeds DPLPMTUD.
- Any other refused datagram counts as sent. To lsquic that is a packet
lost on the wire, so its loss timer paces the retries and a timeout ends
the connection.
- Only `EAGAIN`, `EWOULDBLOCK` and `ENOBUFS` still return short and arm
`WRITABLE`. quinn draws the same line.
- Verified: `fetch-http3-syscall-fault.test.ts`, four new tests, three
fail on stock bun. Also the other h3 suites and `node:quic`, on Linux
and Windows.

### Background
- lsquic calls `us_quic_packets_out` to send a batch of datagrams. One
engine serves every HTTP/3 connection of the process, so one refused
peer stalls the rest.
- A short return with `EAGAIN` stops every send of that engine until
`lsquic_engine_send_unsent_packets` runs. Bun runs it from `on_drain`,
the callback of a writable UDP poll.
- A short return with `EMSGSIZE` retires one datagram and leaves the
engine running.
- Any other errno closes the connection that owns the first unsent
datagram. A pending ICMP error on the shared client socket belongs to no
particular peer, so that does not fit.

<details><summary>Notes</summary>

**Reach: what was run, and what was simulated.** The
unsendable-IP-literal trigger is real (`EACCES` from the kernel, no
privileges needed). `EPERM` from a firewall rule is simulated two ways:
a seccomp filter that fails `sendmsg`/`sendmmsg`, and
`US_FAULT_SENDMSG`. The report this came from also saw it with a real
`iptables -A OUTPUT -p udp --dport <port> -j DROP`. The low egress MTU
is simulated with an `LD_PRELOAD` shim, because the container has no
`CAP_NET_ADMIN` to set a link MTU. No user has reported this. The HTTP/3
API is experimental, so this is a low-priority fix.

**Repro without privileges.** `fetch("https://255.255.255.255/", {
protocol: "http3", signal: AbortSignal.timeout(1500) })`. An IP literal
skips DNS and the connect-time route probe
(`us_quic_socket_context_connect`), and `sendmsg` to the limited
broadcast address fails every time on a socket without `SO_BROADCAST`.
Unset `HTTPS_PROXY` first, a proxy makes fetch refuse HTTP/3 up front.
Stock bun uses 2056 ms of CPU in a 1502 ms wait, and 1354 ms of it after
the abort. With the fix: 69 ms, then 25 ms.

**Tests.**
- `a datagram the kernel refuses to send is dropped without spinning the
HTTP thread` uses that address in a child process and compares
`process.cpuUsage()` with the wall time of a 1 s wait. It needs no fault
injection, so it runs on release builds too. Unfixed: 1365 ms CPU
(release), 1033 ms (debug). Fixed: 34 ms. Skipped on Windows, where the
dual-stack socket sends to `::ffff:255.255.255.255` without an error.
- `Bun.serve: a send error on every datagram does not spin the server's
event loop` arms `EPERM` on every send from inside a request handler and
reads the server child's CPU time. Spin: 2124 ms CPU in 1050 ms. Fixed:
135 ms.
- `Bun.serve: a refused version-negotiation reply does not spin the loop
when no connection exists` sends one long-header datagram with version
`0x0a0a0a0a`, built from a real Initial with its version bytes replaced.
The server queues a Version Negotiation reply for a connection it does
not have. The test first sends the datagram to an unfaulted server and
checks that the reply is a Version Negotiation packet that offers QUIC
v1, so the CPU check cannot pass without a reply to refuse. This is the
only spin with no connection and no timeout to end it. Spin: 2129 ms CPU
in 1052 ms. Fixed: 325 ms against a 150 ms idle baseline for a debug
ASAN server. On the release binary under seccomp the old code uses 2637
ms in 2002 ms.
- `a DPLPMTUD probe above the egress MTU does not stall the transfer`
streams 2.5 s of body under an `LD_PRELOAD` shim that refuses any
datagram above 1400 payload bytes, which is above lsquic's base packet
size and below its first probe. Unfixed: the transfer delivers 32 KB,
stalls for the remaining 11 s, and the next request on the same engine
times out. Fixed: 3.1 MB delivered, no stall, 696 ms CPU in 4049 ms, and
each probe refused once instead of twice.
- The MTU shim declares `sendmmsg` with the flags type of the libc it
builds against: `int` on glibc, `unsigned int` on musl.
- An injected 0 from `US_FAULT_SENDMSG` (the `zero` action) now counts
as one datagram sent in the `sendmmsg` path, as it already does in
`us_quic_send_one`. Before, it left that loop without progress. Only the
test hook can reach this.
- The two `Bun.serve` tests pass on stock bun for a different reason:
the old `sendmmsg` retry does not go through `US_FAULT_SENDMSG`, so an
injected error cannot persist and the retry sends the datagram. This PR
puts the retry behind the hook. The CPU numbers above come from a build
with the hook and the old errno logic.

**`IP_PMTUDISC_PROBE`.** The comment in `us_quic_set_dontfrag` claimed
that this option lets DPLPMTUD probe without `EMSGSIZE`. It does not. It
makes the kernel ignore its cached path MTU. The device MTU still bounds
the datagram: on a 1500 MTU link, a 1472-byte payload is sent and 1473
bytes gives `EMSGSIZE`. The comment now says that, and names where the
errno goes.

**Prior art.** quinn waits for a writable socket only on `WouldBlock`.
It logs every other send error and goes on (`UdpSenderHelper::poll_send`
in `quinn/src/runtime/mod.rs`, with the test
`non_would_block_send_errors_are_logged_and_ignored` using
`PermissionDenied`). Its `quinn-udp` doc says UDP send errors are
non-fatal "because higher-level protocols must employ retransmits and
timeouts anyway".

**Not in this change.** `node:quic`'s own `packets_out`
(`src/runtime/node/quic/endpoint.rs:792-803`) has the same errno
rewrite. It does not spin: `us_udp_socket_send` arms `WRITABLE` only for
`EAGAIN`, so each failure costs one second through lsquic's failsafe. It
also keeps `EMSGSIZE`, as this PR now does. A classifier change there
belongs in its own PR, because oven-sh#40220 rewrites that file (+645/-1056).

**Relation to oven-sh#42678.** Both change the same two `sendmmsg` calls, and
oven-sh#42678 calls this behaviour "a separate bug". This PR is the smaller of
the two, so it should land first. The conflict is mechanical: the call
moves into `us_quic_sendmmsg`.

**Other cases that stay as they are.** `ENOBUFS` remains backpressure,
as oven-sh#35255 chose. That holds on Windows too: `us_quic_send_one` keeps
`WSAENOBUFS` as `ENOBUFS`. Before, it became `EIO`, which the old tail
still treated as backpressure and this PR would have dropped. A UDP
socket that is already closed (`!ls->udp`) still returns short with no
poll to arm, so lsquic's one-second failsafe paces it.

**Self-reviewed:** 8 concerns raised, 5 addressed (the `EMSGSIZE`
classifier and its false comment, the zero-connection spin, the
`node:quic` twin note, the oven-sh#42678 note, the reach note). 3 rejected:
folding the `node:quic` fix in (oven-sh#40220 rewrites that file), patching
lsquic instead (the callback contract has no fourth outcome), and a
client-side address-walk change (a separate behaviour, not this bug).

</details>

<!-- robobun:evidence:begin -->

---

**no test proof** · iteration 0 · platform-specific test(s) that do not
run on this machine, deferring to CI, which covers all platforms:
test/js/web/fetch/fetch-http3-syscall-fault.test.ts

<!-- robobun:evidence:end -->
@robobun

robobun commented Oct 31, 2026

Copy link
Copy Markdown
Collaborator Author

A note for whoever rebases this: #47560 also adds a static bsd_recvmsg_each to packages/bun-usockets/src/bsd.c.

  • There it is the per-message receive loop of macOS below 15.6 (#if LIBUS_UDP_RECV_PER_MESSAGE, which is also true on a fault-injection build). When a recvmsg fails after some datagrams, it records the count in recvbuf->received_before_error and returns -1 with errno, and loop.c delivers those datagrams before it reports the error.
  • The helper in this PR has the same name and another signature, under #if !defined(_WIN32) && !defined(__APPLE__). The two branches merge without a text conflict, and then a Linux fault-injection build has both definitions.
  • return i > 0 ? i : -1 drops the errno of a receive that fails after the first datagram. On a connected socket that errno can be an ICMP error that the kernel does not report again.

So whichever of the two lands second should use the one helper. Two things to know for Linux: LIBUS_UDP_RECV_PER_MESSAGE has to cover the seccomp fallback, and the helper returns 0 for EAGAIN with no datagram, where recvmmsg returns -1, so loop.c would not set recv_would_block_only on that path.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants