Repository navigation
Release 0.7.1: S3 requests outlast 30 s; single PUTs up to 4 GiB - #19
Merged
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixed
request timeout runs from connecting to the end of the response body,
so 0.7.0's single PUTs (up to 1 GiB) and any large GET failed on an
ordinary uplink: at 85 Mbit/s about 320 MB fits in 30 s, less with jobs
sharing it. Every S3 client bigstore builds now allows six hours per
request and keeps the 5-second connect limit, with no per-read limit
(a PUT's response read waits for the whole upload). A caller that builds
its own client for large transfers needs the same.
Changed
backend::SINGLE_PUT_MAX, was 1 GiB),inside the 5 GB that S3 and Wasabi document. Tracking runs write tables
of up to 2 GB; at 1 GiB those went multipart, whose ETag is never the
md5, so every S3 scrub read them back whole. The process-wide budget
for single-PUT bodies held in memory is the same 4 GiB, counted in MiB
units so it fits the semaphore.
Verified against real Wasabi from asset-store (drill, 2026-10-02): a 2,016,058,747-byte run table went up as one PUT with ETag == md5 (0.7.0 sent it as 193 parts and re-read it on every scrub); a no-op health against the peer fell from 355 s to 9 s; 2.2 GB synced in 406 s (0.7.0: 1007 s).