Repository navigation
Conversation
|
Status: draft, stacked on #44680 and #46055. Reproduced on the branch of #44680 with one script on Bun and on Node.js v26.3.0 (built with QUIC). A UDP relay holds the packet of datagram 4 of 10 and delivers it when the client reports a status for that datagram. Bun prints CI does not build the current commit: the branch of #44680 is behind the commit of PR: #46056 |
|
Updated 2:36 AM PT - Oct 22nd, 2026
❌ @robobun, your commit 7c9dbbc has 1 failures in
🧪 To try this PR locally: bunx bun-pr 46056That installs a local version of the PR into your bun-46056 --bun |
lsquic_merge_acks() merges the ACK frames of one batch of packets into one. A range of one frame that lay inside a range of the other cut that range short: [16-8] merged with a saved [15-12][10-9][7-3] gave [16-12][10-9][7-3]. The packets that the newer frame acknowledged in the gaps of the saved one (a late or reordered packet) were not acknowledged, and the send controller then declared them lost. Only a lower end now extends the merged range. The test hands a node:quic client two such ACK frames in one batch and expects every datagram to be acknowledged.
…the packet lsquic reported a DATAGRAM frame lost and then dropped its packet, or cut the frame out of the packet that it sent again. When the peer acknowledged the packet after all, the ACK found no frame to report, so `ondatagramstatus` said 'lost' and never 'acknowledged', and `stats.datagramsAcknowledged` did not count the datagram. Node reports both for one id. The send controller now keeps the (packet number, id) pairs of the last 32 DATAGRAM frames that it reported lost, and compares them with the ranges of each ACK. A match goes to the new `on_datagram_late_ack` hook, which the node:quic shim turns into an 'acknowledged' status. An entry leaves the list when an ACK is 256 packets past it. lsquic keeps nothing for an engine that does not set the hook. lsquic declares losses by other rules than ngtcp2. Its retransmission timers give up on a packet with no ACK of a later packet, where Node reports 'acknowledged' alone. Bun reports 'lost', then 'acknowledged' for such a packet. The comments, the docs and two tests say so.
c9c2097 to
7c9dbbc
Compare
Stacked on #44680 and #46055 (the first commit).
Problem
ondatagramstatusreports only'lost'for a datagram whose packet is declared lost and acknowledged later. Node v26.3.0 reports'lost', then'acknowledged'.send_ctl_handle_regular_lost_packet,patches/lsquic/node-quic-accessors.patch:1872). The late ACK finds no frame.Fix
on_datagram_late_ackhook: node:quic reports'acknowledged'.test/js/node/quic/quic-datagram.test.ts(10 tests fail without the hook), 235 vendoredtest-quic-*files, the HTTP/3 suites.Background
ack_datagramon its ACK.Downsides
datagramsAcknowledgedanddatagramsLost, as in Node.'lost'with no ACK. After a stall Bun reports'lost', then'acknowledged', Node only'acknowledged'. A Bun server can drop a late packet, so two Bun endpoints can still end at'lost'..textgrows 672 bytes.Notes
One script on both runtimes. A UDP relay holds the packet of datagram 4 of 10 and delivers it when the client reports a status for datagram 4. Each row is 3 of 3 runs, release builds, Linux x64. Node.js is v26.3.0 built from source with
--experimental-quic.lost,acknowledgedlostlost,acknowledgedlate-datagram.mjs (run it from the root of the repository)
Where the result is still
'lost'alone. The change is on the sender. It reports'acknowledged'when the peer acknowledges the late packet, and a peer does not acknowledge every late packet.lsquic_rechist_receivedbelow the cutoff thatprocess_ackmoves). This needs no traffic from the receiver: a receiver that sends only ACKs adds a PING every 12 to 27 packets, and the ACK of that PING moves the cutoff. With a server that writes stream data every 2 ms, two Bun endpoints givelostand 10 / 9 / 1 before and after this PR, and the server application never gets datagram 4. A Node.js client against a Bun server gives the same.lost,acknowledgedin 9 of 9 runs when the relay delivers the packet after the tenth datagram, and in 3 of 9 runs when it delivers the packet right at the loss. A Node.js client against that server ends withlostalone in 5 of 6 such runs.process_ack_frame, "Ignore old ack"). So a return path that reorders packets can hide the late ACK.What Node.js does. ngtcp2 keeps the entry of a lost packet with its frames (
rtb_on_pkt_lost). An ACK that covers the entry callsack_datagram(process_acked_pkt). Node.js has no filter for each id (Session::DatagramStatus).Bounds of the list. The list holds the last 32 frames. An ACK whose largest number is more than 256 above the packet removes the frame. Both numbers are choices of this PR. ngtcp2 keeps a lost packet for 3 probe timeouts (
ngtcp2_conn_remove_lost_pkt), and at most the newest max(3, min(256, packets in flight / 2)) lost packets (rtb_detect_lost_pkt, trim). So Bun can report a late'acknowledged'where Node.js has forgotten the packet: with 5 late packets, Node.js reports it for the newest 3 (13 / 11 / 5) and this PR for all 5 (13 / 13 / 5), 3 of 3 runs each. The list is longer than 3 because a retransmission timeout of lsquic reports every datagram in flight at once. The list exists only while theon_datagram_late_ackhook is set. Only the node:quic shim sets it, soBun.serveHTTP/3 andfetchnever allocate it.Order. The late frames of an ACK are reported before the frames that the same ACK acknowledges for the first time, also when one of those is in an older packet. ngtcp2 reports in ascending packet order.
The timers. lsquic and ngtcp2 declare a loss by different rules. lsquic has five: packet threshold, early retransmit, sent time, tail loss probe and retransmission timeout. ngtcp2 has two: packet threshold and time threshold, both after an ACK of a later packet. The two timers of lsquic need no ACK. After a stall of 2 s with no packet lost:
acknowledgedlostlost,acknowledgedEach row is 3 of 3 runs. Two new tests pin the Bun row, one for each timer.
timer-lost.mjs (run it from the root of the repository)
Follow-ups, with no issue yet.
'lost'until an ACK decides. To let the retransmission timeout skip a packet with only DATAGRAM frames does not work: with 10 such packets in flight and no ACK, lsquic's send controller then sends nothing after the timeout (12000 bytes in flight, window 2920, alarm not set). lsquic as it is sends 1 packet for each timeout.'abandoned'is not counted indatagramsLost. WithmaxPendingDatagrams = 2and 3 sends, Node.js reads 0 / 0 / 1 and Bun 0 / 0 / 0. This PR does not touch it.Measurements.
Release flags and ThinLTO. The base is #44680 at 0840e16. The numbers are from the code of this PR before the last change, which touched comments and tests only.
lsquic_send_ctl_got_ack, no lost datagram: 486 -> 489 instructions after quiescence, 492 -> 495 busy (gdb single-step of one call that acknowledges one packet in order). With the hook set and DATAGRAM packets: 528 -> 531 and 534 -> 537.struct lsquic_send_ctl1208 -> 1208 bytes:sc_retx_framesmoves into a 4-byte hole, and the new pointer takes its place and the hole behind it.struct ietf_full_conn10232 -> 10232.struct lsquic_stream_if168 -> 176 (clang record layouts).lsquic_send_ctl_got_ack, the list check true 0 times), 0 in 1,000 lossless datagrams, 1 (one block of 520 bytes) in the reproduction (counters in a debug build).retx_alarm_rings199 -> 199 static instructions. The timeout loop over 8 packets with a retransmittable frame and no DATAGRAM frame: 1,293 -> 1,293 executed (objdump, gdb)..text+672 bytes,.rodata+64 (size -A. The base is a relink of the same objects with the lsquic and the shim of node:quic: report a datagram's status under the id kept in its lsquic frame #44680).lsquic_send_ctl_got_ack0x110f -> 0x1130,send_ctl_handle_regular_lost_packet0x5a1 -> 0x697, newsend_ctl_ack_lost_datagrams0x161 andnq_on_datagram_late_ack0x1e (nm -S). The share of quic: keep every acknowledged packet when lsquic merges two ACK frames #46055 is 20 bytes (lsquic_merge_acks0x1b7 -> 0x1cb).Checks of the list on lsquic's send controller (C harness, ASAN,
SC_SANITY_CHECK): 11 scenarios (loss by ACK evidence and by each timer, packets with and without stream data, several ids in one packet, both early exits oflsquic_send_ctl_got_ack, the cap, the reach). A model check with 200 seeds of 400 steps: 20,000 losses, 11,071 late ACKs, 1,928 ids dropped at the cap, 1,850 by the reach. No difference from the model.Tests.
expect()line, and the other 10 tests of the file pass.remembers the last 32 lost datagrams when a late packet carried 33: without the drop of the oldest entry, ASAN reports a heap-buffer-overflow insend_ctl_keep_lost_datagram.remembers a lost datagram while ACKs that do not cover its packet arrive: fails when such an ACK removes the entry.reports the late acknowledgement once: fails when a reported entry stays in the list.reports a late acknowledgement when no other packet is in flight: fails when the check runs after the exit for an empty queue.arrives latefailed in 4 of 6 debug runs.Suites.
Debug (ASAN) build of the final commit, Linux x64.
test/js/node/quic/: 43 of 43 (quic-datagram20,quic-endpoint8,quic-sni8,quic-stream7).test/js/bun/http/serve-http3.test.ts79 of 79,fetch-http3-client67 of 67,fetch-http3-cold-post2 of 2,fetch-http3-syscall-fault7 of 7.fetch-http3-adversarial: 27 of 27 with a 60 s timeout. The casebody=4194304 > 8 concurrent POST /echotakes 2 to 8 s on debug builds of this PR, of node:quic: report a datagram's status under the id kept in its lsquic frame #44680 and ofmainalike (6 runs each, in turn), so it can pass the default 5 s timeout or not.test-quic-*.mjsfiles, each with the flags of its// Flags:line, on release builds: 235 pass with this PR. With node:quic: report a datagram's status under the id kept in its lsquic frame #44680, 234 pass:test-quic-stream-idle-timeoutfailed once in the parallel run, and passes 5 of 5 alone with both builds.test-quic-session-initial-rttfails with this PR and with node:quic: report a datagram's status under the id kept in its lsquic frame #44680 alike (it expects an RTT under 10 ms).[policy-decision:dep] gate passed · iteration 0 · 7 files touched
fails on main (without fix)
passes on PR (with fix)
diff hotspot
gate history · 1 passed · 0 rejected · iteration 0
evidence per changed file