Skip to content

check: track a set of checked packs for partial checks#9909

Open
mr-raj12 wants to merge 4 commits into
borgbackup:masterfrom
mr-raj12:fix-check-set-of-checked-packs
Open

check: track a set of checked packs for partial checks#9909
mr-raj12 wants to merge 4 commits into
borgbackup:masterfrom
mr-raj12:fix-check-set-of-checked-packs

Conversation

@mr-raj12

@mr-raj12 mr-raj12 commented Jul 14, 2026

Copy link
Copy Markdown
Contributor

The repository check writes a marker holding the last pack id it verified, so a partial check (--max-duration) resumes by skipping every pack up to that marker. This assumes the pack listing is sorted.

Pack files are named by the sha256 of their content, so a pack added between two partial runs can sort anywhere. If its id sorts before the marker, the resumed check skips it and never verifies it.

This replaces the marker with a persisted set of checked pack ids, a HashTableNT mapping pack id to (timestamp, result). A pack is skipped only when it appears in the set recorded as intact; a pack recorded as corrupt is re-verified rather than skipped. So a newly added pack gets checked whatever its name sorts to, and a pack that was corrupt in an earlier run is looked at again. The set is dropped once a cycle has checked every pack.

The set is managed by a PackTracker class. It persists via store.store() (atomic write) with a sha256 appended over the serialized table; on load a blob whose sha256 does not match is discarded, so a rotted set re-checks everything instead of skipping an unverified pack. Ctrl-C is therefore safe: the on-disk set is never partially written and never records a pack that was not verified.

The cycle is checkpointed every 30 minutes rather than every minute: the checkpoint now rewrites the whole table (~41 bytes/pack) instead of a fixed ~70-byte marker, so on large repos each save is no longer O(1). At 30-minute intervals that cost stays negligible while an interrupt still redoes at most 30 minutes of verification work.

The set is stored at cache/checked-packs; docs/internals/data-structures.rst is updated to document it in place of the old marker.

Closes #9897

Replace the single last-pack-checked marker with a persisted set of
checked pack ids, so a pack added between partial runs is verified
regardless of how its content-sha256 name sorts.
@codecov

codecov Bot commented Jul 14, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 84.21053% with 9 lines in your changes missing coverage. Please review.
✅ Project coverage is 85.34%. Comparing base (7da670c) to head (a1b88e2).
⚠️ Report is 5 commits behind head on master.
✅ All tests successful. No failed tests found.

Files with missing lines Patch % Lines
src/borg/repository.py 84.21% 8 Missing and 1 partial ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master    #9909      +/-   ##
==========================================
+ Coverage   85.13%   85.34%   +0.21%     
==========================================
  Files          93       93              
  Lines       15899    15932      +33     
  Branches     2428     2429       +1     
==========================================
+ Hits        13535    13597      +62     
+ Misses       1654     1631      -23     
+ Partials      710      704       -6     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

@ThomasWaldmann ThomasWaldmann left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also: does it handle Ctrl-C correctly?

Comment thread src/borg/testsuite/repository_test.py Outdated
Comment thread src/borg/repository.py Outdated
Comment thread src/borg/repository.py Outdated
Split the vertical import and trim an irrelevant line from the checked-packs comment.
@mr-raj12 mr-raj12 force-pushed the fix-check-set-of-checked-packs branch from b9214b6 to 0e68c7e Compare July 14, 2026 19:44
@mr-raj12

Copy link
Copy Markdown
Contributor Author

Also: does it handle Ctrl-C correctly?

Yes, store.store() writes atomically and appends a sha256 that's verified on load, so an interrupt can't corrupt the set or mark a pack checked when it wasn't. The worst case you redo a last one minute of work, capped at 1 min now that checkpoints run every 60s.

Also shrink the partial-check checkpoint window from 5 minutes to 1 minute
so a Ctrl-C between checkpoints redoes at most a minute of verification work.
@mr-raj12 mr-raj12 force-pushed the fix-check-set-of-checked-packs branch from 0e68c7e to 996331b Compare July 14, 2026 19:52
@ThomasWaldmann

Copy link
Copy Markdown
Member

Docs not updated — data-structures.rst:34 still documents last-pack-checked ("key of last pack checked as text"). It should describe checked-packs instead.

@ThomasWaldmann

Copy link
Copy Markdown
Member

Checkpoint cost grows with repo size (note, not a blocker). The old checkpoint wrote a ~70-byte marker; the new one rewrites the whole table (41 bytes/pack) every 5 minutes — ~4 MB per checkpoint at 100k packs. Fine in practice, just no longer O(1).

Guess we should reduce checkpoint frequency: every 30 mins.

@ThomasWaldmann

Copy link
Copy Markdown
Member

Re-reviewed the updated branch. All 62 repository_test.py tests pass locally, black --check is clean, and I verified the resume semantics directly (see test suggestions below): a pack recorded intact is genuinely skipped on resume, and a full check ignores the recorded set, re-verifies everything, and drops the set.

The update resolves what I'd consider required: the docs are fixed (and better than asked — the old entry was misdocumented under config/, now correctly a cache/ section), and the PackTracker refactor is a clean improvement: state and serialization in one place, check() reads much better, and the class placement and Entry/EntryFormatT pattern match existing conventions (ChunkIndexEntryFormatT in hashindex.pyx).

Remaining minor points, none blocking:

clear() is a mild footgun. It deletes the stored blob but leaves the in-memory table populated. Harmless in today's two call sites (tracker fresh or immediately discarded), but a future caller doing tracker.clear(); tracker.is_intact(...) would get stale answers. Either add self.table.clear() or rename to something like drop_saved() so the name says what it does.

Old marker cleanup. A repo that ran partial checks under older borg keeps its cache/last-pack-checked object forever; nothing deletes it now. A one-line store.delete("cache/last-pack-checked") (ignoring StoreObjectNotFound) in PackTracker.clear() would retire it.

Schema guard on load. If a future borg changes Entry's fields, a valid old blob still loads fine and entry.result then raises AttributeError mid-check — the sha256 protects against rot, not schema evolution. Cheap guard after HashTableNT.read: discard the table unless key_size == 32 and value_type._fields == Entry._fields.

Test gap: the skip itself is untested. The four tests prove the re-check cases, but none proves that an intact-recorded pack is skipped (the point of the mechanism), nor that a full check ignores the set. Both pass on your branch; feel free to take these:

def _spy_hash(repository, monkeypatch):
    hashed_keys = []
    orig_hash = repository.store.hash

    def spy(key):
        hashed_keys.append(key)
        return orig_hash(key)

    monkeypatch.setattr(repository.store, "hash", spy)
    return hashed_keys


def test_check_partial_skips_pack_recorded_intact(tmp_path, monkeypatch):
    # a pack recorded intact this cycle is skipped on resume, not re-verified.
    intact = fchunk(b"INTACT", chunk_id=H(1))
    intact_id = sha256(intact).digest()
    with Repository(str(tmp_path / "repo"), exclusive=True, create=True) as repository:
        pack_key = "packs/" + bin_to_hex(intact_id)
        repository.store_store(pack_key, intact)

        tracker = PackTracker(repository.store)
        tracker.record(intact_id, ok=True)
        tracker.save()

        hashed_keys = _spy_hash(repository, monkeypatch)
        assert repository.check(repair=False, max_duration=3600) is True
        assert pack_key not in hashed_keys  # skipped, not re-verified


def test_check_full_ignores_recorded_set(tmp_path, monkeypatch):
    # a FULL check re-verifies everything even if a partial cycle recorded the pack intact,
    # and drops the set once the cycle is complete.
    intact = fchunk(b"INTACT", chunk_id=H(1))
    intact_id = sha256(intact).digest()
    with Repository(str(tmp_path / "repo"), exclusive=True, create=True) as repository:
        pack_key = "packs/" + bin_to_hex(intact_id)
        repository.store_store(pack_key, intact)

        tracker = PackTracker(repository.store)
        tracker.record(intact_id, ok=True)
        tracker.save()

        hashed_keys = _spy_hash(repository, monkeypatch)
        assert repository.check(repair=False) is True
        assert pack_key in hashed_keys  # full check verified it
        after = PackTracker(repository.store)
        after.load()
        assert len(after) == 0  # set dropped after complete cycle

Tiny wording nits: the docs say "pack key -> timestamp, result" but it's the 32-byte binary pack id ("key" elsewhere in that doc means the store key like packs/<hex>); the PR body says the old checkpoint interval was "every minute" (it was 5 minutes); and the last commit message says "reduce checkpoint interval" where it increases the interval (reduces the frequency).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

borg2 check: move from marker to set-of-checked-packs

2 participants