Harden filesystem publication and noop-wrapped reads - #2716
Open
michaeljohnston-nunkiharmonia wants to merge 3 commits into
Open
Harden filesystem publication and noop-wrapped reads#2716michaeljohnston-nunkiharmonia wants to merge 3 commits into
michaeljohnston-nunkiharmonia wants to merge 3 commits into
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
This comment has been minimized.
This comment has been minimized.
michaeljohnston-nunkiharmonia
force-pushed
the
fix/filesystem-store-publication-upstream
branch
from
August 24, 2026 11:40
2a43b7a to
3ecf9b5
Compare
michaeljohnston-nunkiharmonia
force-pushed
the
fix/filesystem-store-publication-upstream
branch
from
August 24, 2026 11:43
3ecf9b5 to
1595e7e
Compare
michaeljohnston-nunkiharmonia
force-pushed
the
fix/filesystem-store-publication-upstream
branch
from
August 24, 2026 13:37
1595e7e to
0b86377
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What and why
A production Buck2 workload exposed two independent cache-store correctness
failures. Concurrent
FilesystemStoreuploads of the same key could race theeviction-map update and canonical rename, leaving the map populated after the
canonical file disappeared. Separately, a read-only
FastSlowStorewith a noopslow side could reject a valid blob held only by the fast tier of a nested
FastSlowStore, because the outer store gated the read on the inner store'sdurable-only
has()result.This change serializes same-key filesystem publication through a bounded set of
keyed lock shards and lets a new upload repair a stale map entry only when its
canonical file is
NotFound; other I/O errors remain errors. It also delegatesreads directly to the fast store when the slow store advertises
NoopDownloads, since that configuration has no meaningful fallback.There is no upstream issue for this change; it was prompted by the production
failures above.
How was this verified?
The added filesystem regressions cover deterministic repair of a missing
canonical entry and 1,024 synchronized same-digest uploads, followed by a read
of the complete canonical blob. The composed-store regression constructs the
production wrapper shape and reads a blob that exists only in the nested fast
tier. Before the respective fixes, the repair/publication case could lose the
canonical path and the wrapper test returned
Not found in noop store.On
x86_64-unknown-linux-gnuwith nightly 2026-03-24:cargo test -p nativelink-store --test fast_slow_store_test --test filesystem_store_testpassed 28 and 39 tests.
cargo clippy -p nativelink-store --all-targets -- -D warningspassed..rustfmt.tomlpassed for thechanged Rust files.
The coverage lane's target configuration was also exercised directly with
cargo test --target x86_64-unknown-linux-musl --release --locked -p nativelink-store --test filesystem_store_test; all 39 tests passed. This hostdoes not have Nix or Bazel installed, so the repository-wide Nix/Bazel paths
remain delegated to CI.
Equivalent fixes backported to NativeLink 1.6.5 were deployed to the affected
cache. A fresh 2,017-action remote Rust composition, the selected build/tests,
and the final read-only selected-closure probe all passed; the latter had failed
before the composed-store change.
Risk
Both changes are on cache hot paths. The filesystem lock set has 256 shards, so
same-key publication is ordered and unrelated keys that collide on a shard can
briefly serialize; the set is bounded and adds no per-key lifetime state. The
noop-wrapper branch relies on
NoopDownloadsaccurately describing a storethat cannot serve a fallback read; a fast-store error is propagated directly.
There are no config-default, wire-format, public-API, or data-migration changes,
and the patch does not delete cache data. If the publication logic is wrong,
writers or subsequent readers notice first through an upload error or a missing
CAS blob. If the wrapper routing is wrong, read-only cache clients notice first
through a failed CAS read. The production cache and CI cache-route probes are
the first operational detectors.
This change is