Repository navigation
0.3.0: backend::Store and parent-linked folder history - #13
Merged
Merged
Conversation
Store owns the transport (object_store or rclone, private) and offers
head, get, put, put_file, list and download_verified. head replaces
exists; list returns sorted Listed { key, size, modified } (rclone via
lsjson). Every caller (folder, transfer, LFS adapter, CLI) moves over;
the Backend enum and from_config/exists/put_bytes/get_bytes/list/upload/
download/download_verified/rclone_into free functions are gone. The
unverified download is replaced by download_verified in the LFS adapter
and git-mode pull.
A GET the store refuses with S3's InvalidObjectState becomes
backend::Error::Archived { key }, and folder::Error::Archived { key } in
folder mode (pull, log, and the history reads of push and status). Any
other 403 stays object_store's PermissionDenied. object_store keeps the
S3 error code only in the error message, so it is read from there; it
does not expose the storage class at all.
A body that ends before the object's length is now an 'incomplete
download of <key> (<got>/<expected> bytes)' error: Store::get returned
such a body truncated, and download_verified reported it only as an
integrity failure.
cargo deny check failed on the branch base: yoke-derive 0.8.3 (via url, idna and icu_collections) was yanked. Lockfile only.
A version is now a record: a DVC 3 .dvc pointer plus
meta.bigstore {parents, writer, time}, written create-once as
bigstore-history/<key>/<root|p1+p2…>/<id>.dvc, where id is the first
32 hex characters of the SHA-256 of the record's bytes. One listing
gives the graph; the heads are the records no other names as a parent.
Fetching checks the name against the content (id and parents).
0.2 records (<time>-<content id>.dvc) read as a straight line, each
with the id of its file name; they are never written.
The .dvc beside an output records its base (meta.bigstore.base). Push,
before uploading anything: adopts a single head equal to the output
(covering a crash between record and .dvc); otherwise refuses with
Error::StaleBase { base, heads } without a base or when the base is
not the single head, and Error::Diverged { heads } on a fork unless
PushOptions::resolve is Resolve::Merge and the base is a head. Then
objects, manifest, record, and the .dvc last; a relist reports
PushReport::forked_with.
A pull from history writes <into>.dvc with the pulled version as its
base, last, after every file lands. Selector::Latest on a fork is
Diverged; Id matches record ids; AtOrBefore uses each record's time.
SyncState gains Stale and Diverged { heads }. PushOptions::writer
defaults to the host name. The strict .dvc parser accepts exactly
bigstore's two meta shapes and still refuses any other meta.
CLI: folder push --resolve merge; folder log prints parents, writer
and head/fork/merge markers; folder status prints Stale and Diverged.
A .dvc 0.2 wrote (or dvc add) has no meta.bigstore.base. When the output it records equals the single head's output, the folder was last synced to that head, so push treats the head as its base: local edits publish a child of the head instead of StaleBase. The check runs after adopting a head equal to the output and before the StaleBase checks; several heads stay Diverged/StaleBase. status reports LocalAhead to match. A baseless .dvc recording any other content is still StaleBase.
Selector::Id (`--at <hex>`) matches a record id prefix, and now also the content id (md5) in a 0.2 record's name, which 0.2's log printed, so ids noted before the upgrade still select. Both are hex: a prefix matching several versions, of either kind, is AmbiguousId with every candidate listed. 0.3 records are matched by record id only; their content id is not in their name.
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Supersedes #12 (includes its two commits).
Backendenum and its free fns;Archivedonly on S3InvalidObjectState;Record format: records are create-once
bigstore-history/<key>/<parents|root>/<id>.dvc. The id is a fixed-size SHA-256 prefix of the record bytes, andmeta.bigstore {parents, writer, time}sits inside the record.Base tracking: the
.dvcbeside the output carriesmeta.bigstore.base, written last on push and pull. The strict parser accepts only bigstore's meta shapes, so a user'smeta:is still never overwritten.Push order:
.dvcrecords the head's output (the 0.2 upgrade);--resolve merge.Publication runs objects → manifest → record →
.dvc; a relist then reportsforked_with.Pull, status and log: Latest on a fork gives
Diverged; status reports Stale/Diverged;folder logshows the graph.0.2 compatibility: 0.2 records read as a straight line and can still be selected by content id. Hard cutover: 0.2 clients don't see 0.3 records.
Evidence:
.dvcand a merge record.