Skip to content

feat: add check replica command to identify broken snapshot chains - #561

Open
Hassanzadeh-sd wants to merge 1 commit into
longhorn:masterfrom
Hassanzadeh-sd:feat-check-replica-snapshot-chain
Open

feat: add check replica command to identify broken snapshot chains#561
Hassanzadeh-sd wants to merge 1 commit into
longhorn:masterfrom
Hassanzadeh-sd:feat-check-replica-snapshot-chain

Conversation

@Hassanzadeh-sd

@Hassanzadeh-sd Hassanzadeh-sd commented Jul 20, 2026

Copy link
Copy Markdown

Which issue(s) this PR fixes:

Issue longhorn/longhorn#9102

What this PR does / why we need it:

Adds a new longhornctl check replica command that identifies broken snapshot chains in the Longhorn replica data directories, for the v1 data engine.

The remote command follows the existing DaemonSet pattern (same as longhornctl get replica): it deploys privileged pods that run longhornctl-local check replica against the host data directory on each node, then aggregates the per-node JSON results into YAML.

For every replica directory (filterable with --volume-name and --name), the checker:

  • reads volume.meta and every volume-head-*.img.meta / volume-snap-*.img.meta disk metadata file
  • verifies every disk file has a metadata file and vice versa
  • verifies every Parent reference resolves to an existing disk, including snapshot tree branches that are not part of the volume head chain
  • verifies the volume head declared in volume.meta exists, and warns about unexpected extra volume-head-*.img files
  • detects loops in the chain and metadata declaring a mismatching disk name
  • reports the reachable snapshot chain from the volume head to the root for context

A warning is emitted when the replica is in use, since findings may be transient while the engine is modifying the chain (for example during snapshot creation or purge).

Example output for a replica with a missing parent snapshot:

replicas:
  pvc-48a6457d-585e-423b-b530-bbc68a5f948a-0e2603a7:
    - node: ip-10-0-2-123
      directory: /var/lib/longhorn/replicas/pvc-48a6457d-585e-423b-b530-bbc68a5f948a-0e2603a7
      volumeName: pvc-48a6457d-585e-423b-b530-bbc68a5f948a
      snapshotChain:
        - volume-head-001.img
        - volume-snap-40b3b028-b3b3-4a35-a806-8bea77f27c00.img
      errors:
        - 'broken snapshot chain: disk volume-snap-40b3b028-b3b3-4a35-a806-8bea77f27c00.img references parent volume-snap-6f244bbe-2857-46e4-92e2-eb1e16a63ba1.img, but the parent metadata file is missing'

Special notes for your reviewer:

  • The snapshot chain validation logic lives in pkg/local/replica/checker.go (validateSnapshotChain) and is covered by unit tests using synthetic replica directories: healthy chain, snapshot tree branch after revert, missing parent, missing disk file, missing metadata file, missing volume head, chain loop, extra head file, corrupted metadata, and name mismatch.
  • pkg/local/replica/getter.go helper methods getReplicaNamesInDirectory and isReplicaInUse were refactored into package-level functions so the checker can reuse them; no behavior change.
  • This targets the v1 data engine (sparse-file replica directories). The v2 (SPDK) data engine stores snapshots differently and would need a separate implementation.

Additional documentation or context

Add 'longhornctl check replica' which inspects the snapshot chains in
the Longhorn replica data directories on each node and reports
integrity issues:

- snapshots referencing a missing parent (broken chain)
- disk files without metadata files, and metadata files without disk
  files
- missing or corrupted volume.meta and disk metadata files
- a missing volume head, unexpected extra volume head files, and
  metadata declaring a mismatching disk name
- loops in the snapshot chain

The command follows the existing DaemonSet pattern: the remote command
deploys privileged pods that run 'longhornctl-local check replica'
against the host data directory, then aggregates the per-node JSON
results into YAML. Results can be filtered with --volume-name and
--name, and include the reachable snapshot chain from the volume head
for context. A warning is emitted when the replica is in use, since
findings may be transient while the engine is modifying the chain.

Longhorn 9102

Signed-off-by: Sajjad Hassanzadeh <hassanzadeh.sd@gmail.com>
@Hassanzadeh-sd

Copy link
Copy Markdown
Author

Hi @derekbit, this PR is complete from my side and ready for review:

  • CI is green (DCO, commit-lint, build)
  • The branch is rebased on master
  • The snapshot chain validation logic is covered by unit tests (healthy chain, snapshot tree branch after revert, missing parent, missing disk or metadata files, missing volume head, chain loop, extra head file, corrupted metadata, name mismatch)
  • A manual test plan is documented in [IMPROVEMENT] longhornctl supports identifying the broken snapshot chain longhorn#9102, and I am happy to adjust it based on QA feedback

I would appreciate a review when you have time, and if everything looks good, consideration for merging toward the v1.13.0 milestone. I will address any feedback promptly.

Also, as discussed in the issue, feel free to assign longhorn/longhorn#9102 to me. Thanks!

Hassanzadeh-sd added a commit to Hassanzadeh-sd/website that referenced this pull request Aug 9, 2026
Add the new check replica command (longhorn/cli#561, Issue
longhorn/longhorn#9102) to the longhornctl troubleshooting command
list for v1.13.0.

Signed-off-by: Sajjad Hassanzadeh <hassanzadeh.sd@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant