Skip to content

Release 0.7.1: S3 requests outlast 30 s; single PUTs up to 4 GiB - #19

Merged
rmorgans merged 1 commit into
mainfrom
release/0.7.1
Oct 2, 2026
Merged

rmorgans merged 1 commit into
mainfrom
release/0.7.1

Conversation

@rmorgans

@rmorgans rmorgans commented Oct 2, 2026

Copy link
Copy Markdown
Owner

Fixed

  • S3 requests no longer end after 30 seconds. object_store's default
    request timeout runs from connecting to the end of the response body,
    so 0.7.0's single PUTs (up to 1 GiB) and any large GET failed on an
    ordinary uplink: at 85 Mbit/s about 320 MB fits in 30 s, less with jobs
    sharing it. Every S3 client bigstore builds now allows six hours per
    request and keeps the 5-second connect limit, with no per-read limit
    (a PUT's response read waits for the whole upload). A caller that builds
    its own client for large transfers needs the same.

Changed

  • Single PUTs go up to 4 GiB (backend::SINGLE_PUT_MAX, was 1 GiB),
    inside the 5 GB that S3 and Wasabi document. Tracking runs write tables
    of up to 2 GB; at 1 GiB those went multipart, whose ETag is never the
    md5, so every S3 scrub read them back whole. The process-wide budget
    for single-PUT bodies held in memory is the same 4 GiB, counted in MiB
    units so it fits the semaphore.

Verified against real Wasabi from asset-store (drill, 2026-10-02): a 2,016,058,747-byte run table went up as one PUT with ETag == md5 (0.7.0 sent it as 193 parts and re-read it on every scrub); a no-op health against the peer fell from 355 s to 9 s; 2.2 GB synced in 406 s (0.7.0: 1007 s).

@rmorgans
rmorgans merged commit c0da102 into main Oct 2, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant