docs: V2 data engine docs for RDMA transport, per-node SPDK labels, and shallow-copy pipeline

content/docs/1.13.0/advanced-resources/v2-data-engine/_index.md

@@ -1,4 +1,30 @@
 ---
 title: V2 Data Engine
 weight: 110
----
+---
+
+The Longhorn V2 Data Engine is built on the Storage Performance Development Kit (SPDK) and uses NVMe-oF (NVMe over Fabrics) for high-performance, kernel-bypass storage I/O.
+
+## Features
+
+- [Selective V2 Data Engine Activation](./selective-v2-data-engine-activation)
+- [Configurable CPU Cores](./configurable-cpu-cores)
+- [Hugepage Configuration](./hugepage-configuration)
+- [Interrupt Mode](./interrupt-mode)
+- [ublk Frontend Support](./ublk-frontend-support)
+- [RDMA Transport Support](./rdma-transport)
+- [Per-Node V2 Configuration Labels](./node-labels)
+- [Shallow Copy and Deep Copy](./shallow-deep-copy)
+
+## Transport Options
+
+The V2 Data Engine supports two NVMe-oF transport protocols:
+
+- **TCP** (default) — works on any network, no special hardware required
+- **RDMA** — requires RDMA-capable hardware (e.g., Mellanox ConnectX with RoCE v2), provides lower latency and reduced CPU overhead
+
+See [RDMA Transport Support](./rdma-transport) for details on enabling RDMA.
+
+## Efficient Rebuilds
+
+Replica rebuilds use SPDK's shallow copy and range shallow copy primitives, copying only allocated clusters rather than the full volume. See [Shallow Copy and Deep Copy](./shallow-deep-copy) for details.

content/docs/1.13.0/advanced-resources/v2-data-engine/node-labels.md

@@ -0,0 +1,81 @@
+---
+title: Per-Node V2 Configuration Labels
+weight: 3
+---
+
+The Longhorn V2 data engine supports per-node configuration via Kubernetes node labels. These labels allow you to override cluster-wide settings on individual nodes, enabling heterogeneous configurations where different nodes have different CPU, memory, or transport requirements.
+
+## Available Node Labels
+
+| Label | Values | Description |
+|-------|--------|-------------|
+| `node.longhorn.io/nvmf-transport` | `tcp`, `rdma` | NVMe-oF transport type for the instance manager. Set explicitly by the operator. See [RDMA Transport Support](./rdma-transport). |
+| `node.longhorn.io/spdk-cpu-mask` | Hex string (e.g. `0xFF`) | CPU mask for SPDK reactor threads. Overrides the cluster-wide `data-engine-cpu-mask` setting. |
+| `node.longhorn.io/spdk-memory-size` | Decimal MiB (e.g. `16384`) | Hugepage memory size for SPDK in MiB. Overrides the cluster-wide `data-engine-memory-size` setting. |
+| `node.longhorn.io/spdk-interrupt-mode` | `true`, `false` | Enable/disable SPDK interrupt mode. Overrides the cluster-wide `data-engine-interrupt-mode-enabled` setting. Forced to `false` when `nvmf-transport=rdma` (RDMA poll groups cannot use fd-based interrupt wakeup). |
+| `node.longhorn.io/v2-im-cpu-request` | Cores (e.g. `4`) or millicores (e.g. `4000m`) | CPU request for the V2 instance manager pod on this node. Overrides the default CPU request. |
+| `node.longhorn.io/disable-v2-data-engine` | `true`, `false` | Disable the V2 data engine on this node. See [Selective V2 Data Engine Activation](./selective-v2-data-engine-activation). |
+
+## Precedence
+
+Per-node labels take precedence over cluster-wide settings. The resolution order for each setting is:
+
+1. **Per-node label** (highest priority) — if the node has the label, it overrides everything
+2. **Per-IM spec override** — if set on the InstanceManager CR directly
+3. **Cluster-wide setting** (lowest priority) — the global Longhorn setting
+
+The exception is `spdk-interrupt-mode`: if `nvmf-transport=rdma`, interrupt mode is forced to `false` regardless of any other override, because SPDK's RDMA poll groups cannot use fd-based interrupt wakeup.
+
+## Usage Examples
+
+### Enable RDMA on a node
+
+```bash
+kubectl label node <node-name> node.longhorn.io/nvmf-transport=rdma
+```
+
+### Set SPDK CPU mask on a specific node
+
+```bash
+kubectl label node <node-name> node.longhorn.io/spdk-cpu-mask=0xFF
+```
+
+### Set SPDK memory size (16 GiB) on a high-memory node
+
+```bash
+kubectl label node <node-name> node.longhorn.io/spdk-memory-size=16384
+```
+
+### Set V2 IM CPU request to 4 cores on a dedicated storage node
+
+```bash
+kubectl label node <node-name> node.longhorn.io/v2-im-cpu-request=4
+```
+
+### Override interrupt mode on a specific node
+
+```bash
+kubectl label node <node-name> node.longhorn.io/spdk-interrupt-mode=true
+```
+
+### Remove a per-node override (fall back to cluster-wide setting)
+
+```bash
+kubectl label node <node-name> node.longhorn.io/spdk-cpu-mask-
+```
+
+## Verification
+
+Check which labels are applied to a node:
+
+```bash
+kubectl get node <node-name> --show-labels | tr ',' '\n' | grep node.longhorn.io
+```
+
+## When to Use Per-Node Labels
+
+- **Heterogeneous hardware**: Nodes with different CPU counts, memory sizes, or RDMA capabilities can be configured individually
+- **High-memory nodes**: Allocate more hugepages to SPDK on nodes with more RAM
+- **CPU-pinned deployments**: Pin SPDK reactors to specific CPU cores on nodes with dedicated storage CPUs
+- **Mixed transport**: Enable RDMA on nodes with RDMA hardware while leaving TCP-only nodes unchanged
+- **CPU requests**: Allocate more CPU to the instance manager on nodes that host many volumes

content/docs/1.13.0/advanced-resources/v2-data-engine/rdma-transport.md

@@ -0,0 +1,71 @@
+---
+title: RDMA Transport Support
+weight: 1
+---
+
+The Longhorn V2 data engine supports **RDMA (Remote Direct Memory Access)** transport for NVMe-oF replica connections, complementing the default TCP transport. RDMA enables direct memory-to-memory data transfer between the SPDK NVMe-oF target (replica) and initiator (engine) without CPU involvement, reducing latency and CPU overhead for high-throughput workloads.
+
+Phase 1 focuses on **RoCE v2**, an **explicit** per-node transport choice, and a **single transport per engine** for all replica connections during an attachment.
+
+## Prerequisites
+
+- RDMA-capable network hardware (e.g., Mellanox ConnectX-5/6/7 with RoCE v2)
+- RDMA drivers installed on all nodes that will use RDMA transport
+- SPDK built with RDMA support (`--with-rdma`)
+
+## Enabling RDMA Transport
+
+Label each node that should use RDMA:
+
+```bash
+kubectl label node <node-name> node.longhorn.io/nvmf-transport=rdma
+```
+
+To use TCP on a node (default), leave the label unset or set:
+
+```bash
+kubectl label node <node-name> node.longhorn.io/nvmf-transport=tcp --overwrite
+```
+
+Verify labels:
+
+```bash
+kubectl get nodes -o custom-columns=NAME:.metadata.name,TRANSPORT:.metadata.labels.node\.longhorn\.io/nvmf-transport
+```
+
+When a node is labeled `rdma`, the V2 instance manager on that node:
+
+1. Runs with host networking and InfiniBand device access so SPDK can bind the RDMA transport
+2. Creates RDMA NVMe-oF listeners for replicas (and may also advertise a TCP listener for mixed clusters)
+3. Connects engines on that node to **all** replicas using RDMA for that attachment
+
+There is no automatic hardware detection in phase 1 — apply the label only on nodes with working RDMA.
+
+## Mixed Clusters
+
+Longhorn supports mixed clusters where some nodes use RDMA and some use TCP:
+
+- An engine on an RDMA-labeled node uses RDMA for every replica in that attachment
+- An engine on a TCP node uses TCP for every replica in that attachment
+- Replicas on RDMA nodes may advertise both RDMA and TCP addresses so TCP engines can still dial them
+
+This is **not** mid-flight failover between transports, and it is **not** mixing RDMA and TCP within a single engine attachment. Changing a node's transport requires updating the label and reattaching affected volumes.
+
+## Verifying Transport Type
+
+Check the transport-qualified addresses / status for a volume's engine:
+
+```bash
+kubectl get engines.longhorn.io -n longhorn-system <engine-name> -o jsonpath='{.status.replicaStatusMap}' | jq .
+```
+
+## Related Configuration
+
+Per-node SPDK resource overrides (CPU mask, memory size, interrupt mode, IM CPU request) are documented under [Per-Node V2 Configuration Labels](./node-labels). When `nvmf-transport=rdma`, interrupt mode is forced off.
+
+## Limitations
+
+- RDMA transport is only available for the V2 data engine
+- Dynamic transport switching for running volumes is not supported — volumes must be detached and reattached after changing the node label
+- iWARP and non-RoCE RDMA protocols are not supported (RoCE v2 only)
+- Auto-detection of RDMA hardware is not part of phase 1

content/docs/1.13.0/advanced-resources/v2-data-engine/shallow-deep-copy.md

@@ -0,0 +1,43 @@
+---
+title: Shallow Copy and Deep Copy
+weight: 4
+---
+
+The Longhorn V2 data engine uses SPDK's **shallow copy** and **deep copy** primitives for efficient replica rebuilds and data synchronization. These copy mechanisms operate at the cluster level, copying only allocated data and skipping unallocated (zero) regions.
+
+## Shallow Copy
+
+Shallow copy copies only the clusters that the source lvol has allocated, writing them onto the destination bdev. Unallocated clusters in the source are skipped — the destination keeps whatever it already had there.
+
+### Pipelined Shallow Copy
+
+Shallow copy can be pipelined with a configurable depth, allowing concurrent in-flight copy operations. The depth is controlled by the `data-engine-shallow-copy-pipeline-depth` setting:
+
+```json
+{"v2": "4"}
+```
+
+Default depth is 1 (sequential). Higher values allow more concurrent copy operations, speeding up rebuilds on storage backends that can handle parallel I/O.
+
+### Range Shallow Copy
+
+Range shallow copy copies a specific list of clusters and unmaps (TRIMs) all others on the destination. This is used when the destination already has a partial copy of the data — only the mismatching clusters need to be copied, and the rest are unmapped to ensure consistency.
+
+Range shallow copy is the mechanism used for incremental rebuilds when the destination already has most of the data and only a subset of clusters have changed.
+
+## Deep Copy
+
+Deep copy copies all allocated clusters from the source lvol **and its snapshot ancestors** to a destination bdev. Unlike shallow copy, deep copy reads through the entire snapshot chain, duplicating all data.
+
+Deep copy is used when a complete, independent copy of the data is needed (e.g., for volume cloning or backup operations).
+
+## Rebuild Usage
+
+During a v2 volume rebuild:
+1. The engine creates a rebuild snapshot on the source replica
+2. The destination head lvol is cloned from the rebuild snapshot (thin clone — no data copied yet)
+3. For each snapshot in the chain, shallow copy copies the allocated clusters from source to destination
+4. If the destination already has an intact snapshot (from a previous partial rebuild), it is reused — no copy needed
+5. If the destination has a corrupted but range-checksum-eligible snapshot, range shallow copy copies only the mismatching clusters
+
+This cluster-level copy approach means that rebuild time and space usage are proportional to the **actual data size**, not the volume's logical size.

content/docs/1.13.0/references/settings.md

@@ -118,6 +118,7 @@ weight: 1
   - [Data Engine Hugepage Enabled](#data-engine-hugepage-enabled)
   - [Data Engine Memory Size](#data-engine-memory-size)
   - [Data Engine Interrupt Mode Enabled](#data-engine-interrupt-mode-enabled)
+  - [Data Engine Shallow Copy Pipeline Depth](#data-engine-shallow-copy-pipeline-depth)
   - [Log Path](#log-path)
   - [Snapshot Heavy Task Concurrent Limit](#snapshot-heavy-task-concurrent-limit)
   - [System Managed CSI Components Resource Limits](#system-managed-csi-components-resource-limits)
@@ -1300,6 +1301,21 @@ Controls whether the Storage Performance Development Kit (SPDK) target daemon ru
 > **Warning**
 > - DO NOT CHANGE THIS SETTING WITH ATTACHED VOLUMES. Longhorn will block this setting update when there are attached v2 volumes.
 
+
+#### Data Engine Shallow Copy Pipeline Depth
+
+> Default: `{"v2":"1"}`
+
+Applies only to the **V2 Data Engine**.
+
+Controls the pipeline depth for shallow copy operations during replica rebuilds. Higher values allow more concurrent in-flight copy operations, potentially speeding up rebuilds on storage backends that can handle parallel I/O.
+
+- `1` (default): Sequential copy (no pipelining)
+- Higher values (e.g. `4`): Allow concurrent copy operations
+
+> **Warning**
+> - Higher pipeline depths increase memory usage proportional to the depth × cluster size. Monitor memory consumption when increasing this value.
+
 #### Log Path
 
 > Default: `/var/lib/longhorn/logs/`