feat: encryption at rest for dm-thin volumes (LUKS2/dm-crypt)

[abonillabeeche]

Draft — stacked on #64 and #66 (both unmerged). This branch is based on top of
#64 (LVM volume/snapshot lifecycle hardening) and the #66 snapshot record-location
work, so the diff below currently includes those commits. For review, focus on the
single feat: encryption at rest… commit (pkg/lvm/encryption.go + the encryption
hunks). Once #64/#66 land in main I'll rebase so the diff is encryption-only.
@WebberHuang1118 — I'd value your steer on sequencing and on the RBAC question below.

What this adds

Opt-in encryption at rest for dm-thin LVM volumes using LUKS2 / dm-crypt.

Encryption is declared on the StorageClass (encrypted: "true") and follows the exact
same CSI-secret + CRYPTO_KEY_* convention Longhorn uses, so the existing Harvester
webhook and dashboard accept it unchanged:

Secret key	Meaning	Default
CRYPTO_KEY_VALUE	passphrase (required)	—
CRYPTO_KEY_CIPHER	cipher	aes-xts-plain64
CRYPTO_KEY_HASH	hash	sha256
CRYPTO_KEY_SIZE	key size	256
CRYPTO_PBKDF	PBKDF	argon2i

Each volume is LUKS2-formatted and opened as a dm-crypt device
(/dev/mapper/csi-lvm-<volID>) at NodePublishVolume; mount / bind-mount / expand all
operate through the mapper. The passphrase is passed to cryptsetup over stdin, never
via argv. Unpublish closes the mapper; expand resizes the LUKS device first.

RBAC design question for @WebberHuang1118

The external-provisioner sidecar (controller SA) must resolve the
csi.storage.k8s.io/provisioner-secret-name/-namespace referenced by an encrypted SC.
Because the recommended SC uses ${pvc.namespace} templating, the secret can live in
any namespace — so I added cluster-wide secrets: get/list/watch to the
controller ClusterRole (this mirrors what Longhorn's chart grants).

Questions:

1. Are you comfortable with cluster-wide secret read on the controller SA (the standard
   external-provisioner pattern), or would you prefer to constrain encryption to a
   fixed secret namespace with a namespaced Role (loses per-namespace key
   isolation and diverges from the Harvester UI's ${pvc.namespace} convention)?
2. The node plugin does not need any new RBAC — the node-publish-secret is
   delivered to it by kubelet, not fetched via the API. Flagging in case you'd rather be
   explicit about it.

QA — validated end-to-end on a live Harvester v1.8.1 cluster

Check	Result
On-disk encryption	✅ raw LV shows LUKS magic + luksDump = LUKS2 / aes-xts-plain64; dm-crypt mapper active
Deploy image into encrypted SC	✅ via CDI host-assisted copy clone (writes through the LUKS mapper)
Cross-SC clone (unencrypted image → encrypted SC)	✅ requires cloneStrategy=copy (set via SC annotation in the example)
Linux VM on encrypted SC	✅ boots, runs
Windows VM on encrypted SC	✅ boots, runs
Volume expand (online)	✅ LUKS device resized through the mapper
VolumeSnapshot of encrypted volume	✅ captures ciphertext at the dm-thin layer below dm-crypt
Kasten K10 backup + restore	✅ restored VM's volume verified still LUKS2-encrypted after restore

Benchmark (fio, same node, encrypted vs plain — AES-NI)

Profile	Plain	Encrypted
seqwrite 1M qd16	6497 IOPS / 6498 MiB/s	6424 IOPS / 6424 MiB/s
randread 4k qd32	75.3k IOPS / 294 MiB/s	74.5k IOPS / 291 MiB/s
randwrite 4k qd32	51.1k IOPS / 200 MiB/s	58.3k IOPS / 228 MiB/s
mixed 4k qd32 (r/w)	48.0k / 20.6k IOPS	47.7k / 20.5k IOPS

Overhead is within run-to-run noise on this hardware (AES-NI). Encryption is a per-SC
opt-in, so unencrypted volumes are unaffected.

Notes for reviewers

• Kasten K10 block-mode export re-publishes the snapshot-clone through the CSI node
  path, which luksOpens the device — so an exported copy is readable plaintext at
  export time (expected; K10 re-encrypts at its own rest layer). Flagging as a nuance,
  not a defect.
• A companion Harvester dashboard PR exposes the encryption toggle + secret picker in
  the LVM StorageClass form (the only UI touchpoint that is provisioner-specific; images,
  volumes, VMs and restore already inherit encryption via the SC).

Test plan

• Unit tests for CRYPTO_KEY_* parsing + cryptsetup argument construction (pkg/lvm/encryption_test.go); gofmt clean.
• Live QA matrix above.

Related

• Tracking issue: [FEATURE] Encryption at rest for LVM local volumes (LUKS2/dm-crypt) harvester#11442
• Dashboard UI: feat(storage): expose volume encryption for the LVM CSI StorageClass harvester-ui-extension#1121
• Docs: docs(lvm): document encryption at rest for LVM volumes docs#1107

[abonillabeeche]

Evidence that IO is routed through the dm-crypt layer

Re: the concern about whether the volume's IO actually passes through the encryption layer (vs. a LUKS header that's present but bypassed). Here's a plaintext-in / ciphertext-on-disk differential run on a live cluster.

Setup: a plain pod wrote a known marker (DMCRYPT_ROUTING_PROOF_2026, repeated) to a Block-mode PVC on an encrypted dm-thin StorageClass, then the same region was read two ways on the node.

1. The workload's device is a LUKS2 crypt target sitting on the raw LV

# cryptsetup status csi-lvm-<vol>
  type:    LUKS2
  cipher:  aes-xts-plain64   keysize: 256
  device:  /dev/mapper/vmvg-<vol>      # underlying raw LV
  offset:  16777216 bytes             # 16 MiB LUKS2 header

# dmsetup table csi-lvm-<vol>
  0 2064384 crypt aes-xts-plain64 :32:logon:... 0 254:30 32768   # 'crypt' target over the LV (dm 254:30)

2. Differential — same bytes, read below the crypt layer vs through the mapper

Read path	marker hits
Raw LV /dev/mapper/vmvg-<vol> (below dm-crypt), 256 MiB scanned	0
Mapper /dev/mapper/csi-lvm-<vol> (what the workload uses), 32 MiB scanned	38784

First data sector of the raw LV is high-entropy (8c 8e 3c 72 28 da 8c 61 e5 3c cf d7 …), not the marker.

The plaintext is visible only through the dm-crypt mapper; on the backing LV it is ciphertext. That is only possible if every read/write the workload issues traverses the crypt target — i.e. IO is genuinely routed through the encryption layer.

Reproduce (any encrypted PVC, on its node):

MAP=/dev/mapper/csi-lvm-<volumeHandle>
RAW=$(cryptsetup status $MAP | awk '/device:/{print $2}')
# write a marker through the mapper (e.g. from a pod using the PVC), then:
dd if=$RAW bs=1M count=256 | grep -a -c MARKER   # -> 0   (ciphertext on disk)
dd if=$MAP bs=1M count=32  | grep -a -c MARKER   # -> >0  (plaintext via crypt)

[abonillabeeche]

Boot-time & power-cycle testing: encrypted vs plain VM

Follow-up to the fio benchmark and the dm-crypt IO-path proof above. Both VMs are identical Ubuntu 24.04 (noble) guests scheduled on the same node from the same base image; the only difference is the StorageClass (lvm-thin-encrypted, LUKS2/aes-xts-plain64 vs plaintext lvm-thin), so this isolates the encryption cost.

Guest boot time (systemd-analyze time)

VM	Kernel	Userspace	Total
encrypted	4.389s	6.833s	11.223s
plain	4.449s	6.650s	11.100s
encrypted (post-reboot re-check)	4.394s	7.002s	11.397s

Difference is ~1%, within run-to-run noise. Top systemd-analyze blame units are identical on both (systemd-networkd-wait-online, cloud-config, snapd). This is expected: dm-crypt runs on the host (luksOpen at NodeStage, transparent block layer), so the guest sees a plain vda and its boot path is unaffected. lsblk inside both guests confirms an ordinary vda with no crypt device visible in-guest.

Off/On and reboot (wall-clock, virtctl start/restart → VMI Ready)

Operation	encrypted	plain
Off → On (stop/start)	10s	8s
Reboot #1	11s	6s
Reboot #2	11s	8s

Every cycle succeeded; both VMs returned to Running/Ready reliably. The encrypted VM is consistently ~2-4s slower at the infra level — that gap is the host-side cryptsetup luksOpen (Argon2i PBKDF) at attach time, a fixed one-time cost per attach, not a per-IO tax.

Log health

• 0 failed systemd units on both VMs.
• Only benign, identical warnings on both (pam_lastlog.so dlopen — a known Ubuntu 24.04 packaging quirk — and the ITS mitigation kernel notice). No storage, dm-crypt, or filesystem errors.

Summary

Encryption at rest adds negligible guest-boot overhead (~1%) plus a small fixed attach-time cost (~2-4s) for the LUKS unlock. Stop/start and reboot are robust across repeated cycles. Consistent with the fio result (~1% throughput overhead via AES-NI): encryption is effectively free at runtime, the only measurable cost being the one-time unlock at volume attach.

[abonillabeeche]

Disk performance: encrypted vs. plain (and a --perf-no_read_workqueue experiment)

Following the boot-time validation above, here is a disk-throughput characterization of an encrypted (LUKS2/dm-crypt) volume vs. a plain one, plus an experiment with cryptsetup's --perf-no_read_workqueue activation flag to try to close the read gap.

Method

• Single-node cluster, one guest at a time (twins stopped to remove thin-pool contention). Windows Server 2022, virtio-blk, 4 vCPU / 16 GiB.
• Backing: lvm-thin (plain) vs. lvm-thin-encrypted (LUKS2, aes-xts-plain64 / sha256 / 256-bit / argon2i) — same VG/thin pool, RWO.
• diskspd 2.2.0, 60 s per case, 50 GiB test file on a dedicated NTFS data disk (F:).
• Cases: seq-*-1M-QD32, rand-*-4K-QD32x4 (128 outstanding), rand-4K-QD32x4-70r30w.
• Columns: IOPS, MB/s, avg latency (ms). Multiple samples taken because a shared thin pool makes single 60 s runs noisy for the sequential/write cases; 4K random-read was the clean, reproducible signal.

Results — 4K random read, QD32×4 (the clean signal)

Config	IOPS	MB/s	avg lat (ms)
Plain (unencrypted)	~140,500 / ~141,300	~549 / ~552	0.89 / 0.87
Encrypted, default (queued kcryptd read)	~71,400 / ~71,200	~279 / ~278	1.78 / 1.78
Encrypted, --perf-no_read_workqueue	66,076 / 51,697	258 / 202	1.91 / 2.45

Encrypted 4K random-read runs at roughly half the plain IOPS at this queue depth. Writes and sequential transfers were within run-to-run noise between plain and encrypted (encrypted 4K random-write ~14.7–15.7k IOPS vs plain ~12–21k across samples; both sequential cases dominated by thin-pool variance).

The --perf-no_read_workqueue experiment

Hypothesis: at high read IOPS the bounded kcryptd read workqueue — not AES-NI throughput — is the limiter, so activating the LUKS device with --perf-no_read_workqueue (inline decryption on the submitting task) should recover most of the gap. It's an activation-time flag only, no on-disk format change.

I built a driver image that adds the flag to cryptsetup luksOpen, deployed it, and confirmed it was live (the dm-crypt table showed ... 1 no_read_workqueue on the benchmarked device). Result: it did not help — it slightly hurt. Two samples gave 66,076 and 51,697 IOPS, at or below the stable ~71k of the default queued path, versus ~140k plain.

Why (interpretation): with only 4 vCPU and QD32×4 (128 outstanding I/Os), the bottleneck is CPU, not the read workqueue. Inline decryption serializes the decrypt onto the submitting CPUs, whereas the default path fans decryption across kcryptd workers — so removing the workqueue reduces available parallelism and throughput goes down. The read penalty is inherent dm-crypt cost at high IOPS on a low-vCPU guest, not a workqueue-scheduling artifact.

Decision: the flag is not included in this PR (reverted). A Linux guest showed only ~1% impact earlier because fio there drove ~75k IOPS — below the contention point where the penalty appears.

Takeaway

Encryption-at-rest here is effectively free for boot, sequential, and write workloads, but costs roughly half the IOPS on high-queue-depth small random reads on low-vCPU guests. This is expected dm-crypt behavior and is documented here rather than tuned away. Guests that need maximum small-random-read IOPS should be sized with more vCPUs or use a non-encrypted StorageClass.

[abonillabeeche]

Pushed a follow-up fix commit (5af9aac) covering three issues found while running the storage-validator against an encrypted dm-thin StorageClass:

• LUKS resize passphrase — cryptsetup resize blocked on an interactive prompt during NodeExpandVolume because the volume key isn't reachable in the node plugin's keyring; the passphrase is now fed on stdin via --key-file - (never argv). Requires the StorageClass to wire a node-expand-secret so external-resizer populates the request secrets (a clear InvalidArgument is returned if it's missing).
• LUKS2 header capacity — the default LUKS2 16 MiB data offset made the dm-crypt mapper expose 16 MiB less than the backing LV, so exact-fit consumers (CDI/KubeVirt image imports) failed. The backing LV is now grown by that overhead at create and expand; reported (usable) capacity is unchanged.
• Secret logging — NodeExpandVolume request secrets are redacted with protosanitizer.StripSecrets.

Validated end-to-end: a full storage-validator run against an encrypted LVM CSI StorageClass on single-node Harvester v1.8.1 passed all checks (create/use, offline expand, snapshot, image, VM boot, hotplug). The validator needed RWO/single-node support for this, sent separately as harvester/storage-validator#61.