docs(concepts): add Instance Manager section

content/docs/1.11.0/concepts.md

@@ -15,10 +15,11 @@ For the installation requirements, go to [this section.](../deploy/install/#inst
 
 - [1. Design](#1-design)
   - [1.1. The Longhorn Manager and the Longhorn Engine](#11-the-longhorn-manager-and-the-longhorn-engine)
-  - [1.2. Advantages of a Microservices Based Design](#12-advantages-of-a-microservices-based-design)
-  - [1.3. CSI Driver](#13-csi-driver)
-  - [1.4. CSI Plugin](#14-csi-plugin)
-  - [1.5. The Longhorn UI](#15-the-longhorn-ui)
+  - [1.2. The Instance Manager](#12-the-instance-manager)
+  - [1.3. Advantages of a Microservices-Based Design](#13-advantages-of-a-microservices-based-design)
+  - [1.4. CSI Driver](#14-csi-driver)
+  - [1.5. CSI Plugin](#15-csi-plugin)
+  - [1.6. The Longhorn UI](#16-the-longhorn-ui)
 - [2. Longhorn Volumes and Primary Storage](#2-longhorn-volumes-and-primary-storage)
     - [2.1. Thin Provisioning and Volume Size](#21-thin-provisioning-and-volume-size)
     - [2.2. Reverting Volumes in Maintenance Mode](#22-reverting-volumes-in-maintenance-mode)
@@ -73,7 +74,28 @@ In the figure below,
 
 {{< figure alt="read/write data flow between the volume, controller instance, replica instances, and disks" src="/img/diagrams/architecture/how-longhorn-works-with-kubernetes.svg" >}}
 
-## 1.2. Advantages of a Microservices Based Design
+## 1.2. The Instance Manager
+
+The [Instance Manager](https://github.com/longhorn/longhorn-instance-manager) is the per-node component that hosts and manages the lifecycle of engine and replica instances. It runs as a pod in the `longhorn-system` namespace, and is created and supervised by the Longhorn Manager. Unlike the Longhorn Manager, which is a single DaemonSet across the cluster, the Instance Manager is a system-managed component whose lifecycle is owned by Longhorn itself.
+
+When the Longhorn Manager decides to attach a volume, it does not start the engine or replica processes directly. Instead, it instructs the Instance Manager on the relevant node to start them inside the Instance Manager pod. Each worker node runs a single Instance Manager pod per data engine version, and that pod hosts the engine and replica instances for many volumes that land on the node. For a given volume, one engine instance lives in the Instance Manager on the node where the workload Pod runs, and one replica instance lives in the Instance Manager on each node selected for that volume's replicas. The number of replicas per volume is controlled by the [Default Replica Count](../references/settings/#default-replica-count) setting and can be overridden per volume.
+
+> Note: For RWX volumes without the `migratable` flag, the engine runs on the node hosting the share-manager pod rather than on the workload node.
+
+Instance Managers also act as the gate between Longhorn's control plane and data plane. Each Instance Manager pod runs a proxy service that the Longhorn Manager uses to reach the hosted engine and replica instances, so control-plane operations (attach/detach, snapshot, backup, replica rebuild) flow through this proxy. When a [Storage Network](../advanced-resources/deploy/storage-network/#setting-storage-network-during-longhorn-installation) is configured, Instance Manager pods also route their traffic through it.
+
+The hosting model differs between data engines:
+
+- **V1 Data Engine.** The Instance Manager runs each engine and each replica as a Linux process inside the pod. The engine process is also what exposes the volume's block device to the host, using iSCSI as the frontend. A single V1 Instance Manager pod can host engine and replica processes for many volumes. Because engine and replica processes share the pod, the Instance Manager's resource consumption scales with the aggregate I/O load of the volumes hosted on the node. Review the [Guaranteed Instance Manager CPU](../references/settings/#guaranteed-instance-manager-cpu) setting before scaling replicas or attaching high-throughput volumes on a node.
+- **V2 Data Engine.** The Instance Manager runs an SPDK target process (`spdk_tgt`) inside the pod, and SPDK takes over the full storage path, including the disks themselves. Each V2 block-type disk is imported into the target as a Logical Volume Store (LVS), and replicas live on top as SPDK logical volume bdevs. Engines are exposed as SPDK RAID block devices built from those replicas. The frontend presenting the block device to the host (NVMe-TCP or UBLK) is also driven from this Instance Manager. Because `spdk_tgt` runs in polling mode by default, the V2 Instance Manager reserves dedicated CPU cores and memory (hugepages when enabled) on each node. Use the [Data Engine CPU Mask](../references/settings/#data-engine-cpu-mask) and [Data Engine Memory Size](../references/settings/#data-engine-memory-size) settings to tune these reservations. Longhorn also supports [Interrupt Mode](../advanced-resources/v2-data-engine/interrupt-mode/) as an alternative when reducing CPU consumption is more important than raw I/O performance.
+
+If both data engines are enabled on a cluster, each node runs two Instance Manager pods (one per data engine version), each with its own CPU reservation. During a Longhorn upgrade, a new Instance Manager pod is created alongside the existing one on each affected node. The old pod keeps hosting the engine and replica instances that are already running so that live volumes stay online, while newly created and newly attached volumes land on the upgraded pod. Existing volumes only move to the new Instance Manager when they are detached and reattached (typically as part of the engine upgrade workflow), and the old pod is only removed once no instances remain inside it. Because both the old and new pods keep their CPU and memory reservations during this window, each node needs enough spare capacity to run the extra pods until the upgrade completes.
+
+Because the engine and replica instances live inside the Instance Manager pod, the Instance Manager defines the failure domain for everything it hosts. If an Instance Manager pod is restarted or evicted, every engine and replica instance it was hosting goes with it, and the Longhorn Manager must reattach the affected volumes and rebuild replicas as needed. For this reason, Longhorn reserves CPU for the Instance Manager pod (see [Guaranteed Instance Manager CPU](../best-practices/#guaranteed-instance-manager-cpu)) and avoids restarting it while it is still hosting active instances. For the same reason, Longhorn protects the Instance Manager pod from accidental eviction or drain with a PodDisruptionBudget, whose behavior is controlled by the [Node Drain Policy](../references/settings/#node-drain-policy) setting.
+
+This also means that the Instance Manager is the pod you target when debugging engine or replica behavior — there is no separate per-volume engine pod. Instance Manager pods follow the naming pattern `instance-manager-<hash>` and carry the `longhorn.io/node=<node-name>` label, which is the practical entry point for `kubectl logs` and `kubectl exec` when troubleshooting volume I/O issues on a specific node.
+
+## 1.3. Advantages of a Microservices-Based Design
 
 In Longhorn, each Engine only needs to serve one volume, simplifying the design of the storage controllers. Because the failure domain of the controller software is isolated to individual volumes, a controller crash will only impact one volume.
 
@@ -83,14 +105,14 @@ Because each volume has its own controller, the controller and replica instances
 
 Longhorn can create a long-running job to orchestrate the upgrade of all live volumes without disrupting the on-going operation of the system. To ensure that an upgrade does not cause unforeseen issues, Longhorn can choose to upgrade a small subset of the volumes and roll back to the old version if something goes wrong during the upgrade.
 
-## 1.3. CSI Driver
+## 1.4. CSI Driver
 
 The Longhorn CSI driver takes the block device, formats it, and mounts it on the node. Then the [kubelet](https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet/) bind-mounts the device inside a Kubernetes Pod. This allows the Pod to access the Longhorn volume.
 
 The required Kubernetes CSI Driver images will be deployed automatically by the longhorn driver deployer.
 To install Longhorn in an air gapped environment, refer to [this section](../deploy/install/airgap).
 
-## 1.4. CSI Plugin
+## 1.5. CSI Plugin
 
 Longhorn is managed in Kubernetes via a [CSI Plugin.](https://kubernetes-csi.github.io/docs/) This allows for easy installation of the Longhorn plugin.
 
@@ -106,7 +128,7 @@ In contrast, v2 volumes come with different prerequisites, depending on the conf
 - For the NVMe-TCP frontend, the `nvme_tcp` module is necessary.
 - For the UBLK frontend, both the `ublk_drv` module and huge pages support must be enabled.
 
-## 1.5. The Longhorn UI
+## 1.6. The Longhorn UI
 
 The Longhorn UI interacts with the Longhorn Manager through the Longhorn API, and acts as a complement of Kubernetes. Through the Longhorn UI, you can manage snapshots, backups, nodes and disks.
 

content/docs/1.11.1/concepts.md

@@ -15,10 +15,11 @@ For the installation requirements, go to [this section.](../deploy/install/#inst
 
 - [1. Design](#1-design)
   - [1.1. The Longhorn Manager and the Longhorn Engine](#11-the-longhorn-manager-and-the-longhorn-engine)
-  - [1.2. Advantages of a Microservices Based Design](#12-advantages-of-a-microservices-based-design)
-  - [1.3. CSI Driver](#13-csi-driver)
-  - [1.4. CSI Plugin](#14-csi-plugin)
-  - [1.5. The Longhorn UI](#15-the-longhorn-ui)
+  - [1.2. The Instance Manager](#12-the-instance-manager)
+  - [1.3. Advantages of a Microservices-Based Design](#13-advantages-of-a-microservices-based-design)
+  - [1.4. CSI Driver](#14-csi-driver)
+  - [1.5. CSI Plugin](#15-csi-plugin)
+  - [1.6. The Longhorn UI](#16-the-longhorn-ui)
 - [2. Longhorn Volumes and Primary Storage](#2-longhorn-volumes-and-primary-storage)
     - [2.1. Thin Provisioning and Volume Size](#21-thin-provisioning-and-volume-size)
     - [2.2. Reverting Volumes in Maintenance Mode](#22-reverting-volumes-in-maintenance-mode)
@@ -73,7 +74,28 @@ In the figure below,
 
 {{< figure alt="read/write data flow between the volume, controller instance, replica instances, and disks" src="/img/diagrams/architecture/how-longhorn-works-with-kubernetes.svg" >}}
 
-## 1.2. Advantages of a Microservices Based Design
+## 1.2. The Instance Manager
+
+The [Instance Manager](https://github.com/longhorn/longhorn-instance-manager) is the per-node component that hosts and manages the lifecycle of engine and replica instances. It runs as a pod in the `longhorn-system` namespace, and is created and supervised by the Longhorn Manager. Unlike the Longhorn Manager, which is a single DaemonSet across the cluster, the Instance Manager is a system-managed component whose lifecycle is owned by Longhorn itself.
+
+When the Longhorn Manager decides to attach a volume, it does not start the engine or replica processes directly. Instead, it instructs the Instance Manager on the relevant node to start them inside the Instance Manager pod. Each worker node runs a single Instance Manager pod per data engine version, and that pod hosts the engine and replica instances for many volumes that land on the node. For a given volume, one engine instance lives in the Instance Manager on the node where the workload Pod runs, and one replica instance lives in the Instance Manager on each node selected for that volume's replicas. The number of replicas per volume is controlled by the [Default Replica Count](../references/settings/#default-replica-count) setting and can be overridden per volume.
+
+> Note: For RWX volumes without the `migratable` flag, the engine runs on the node hosting the share-manager pod rather than on the workload node.
+
+Instance Managers also act as the gate between Longhorn's control plane and data plane. Each Instance Manager pod runs a proxy service that the Longhorn Manager uses to reach the hosted engine and replica instances, so control-plane operations (attach/detach, snapshot, backup, replica rebuild) flow through this proxy. When a [Storage Network](../advanced-resources/deploy/storage-network/#setting-storage-network-during-longhorn-installation) is configured, Instance Manager pods also route their traffic through it.
+
+The hosting model differs between data engines:
+
+- **V1 Data Engine.** The Instance Manager runs each engine and each replica as a Linux process inside the pod. The engine process is also what exposes the volume's block device to the host, using iSCSI as the frontend. A single V1 Instance Manager pod can host engine and replica processes for many volumes. Because engine and replica processes share the pod, the Instance Manager's resource consumption scales with the aggregate I/O load of the volumes hosted on the node. Review the [Guaranteed Instance Manager CPU](../references/settings/#guaranteed-instance-manager-cpu) setting before scaling replicas or attaching high-throughput volumes on a node.
+- **V2 Data Engine.** The Instance Manager runs an SPDK target process (`spdk_tgt`) inside the pod, and SPDK takes over the full storage path, including the disks themselves. Each V2 block-type disk is imported into the target as a Logical Volume Store (LVS), and replicas live on top as SPDK logical volume bdevs. Engines are exposed as SPDK RAID block devices built from those replicas. The frontend presenting the block device to the host (NVMe-TCP or UBLK) is also driven from this Instance Manager. Because `spdk_tgt` runs in polling mode by default, the V2 Instance Manager reserves dedicated CPU cores and memory (hugepages when enabled) on each node. Use the [Data Engine CPU Mask](../references/settings/#data-engine-cpu-mask) and [Data Engine Memory Size](../references/settings/#data-engine-memory-size) settings to tune these reservations. Longhorn also supports [Interrupt Mode](../advanced-resources/v2-data-engine/interrupt-mode/) as an alternative when reducing CPU consumption is more important than raw I/O performance.
+
+If both data engines are enabled on a cluster, each node runs two Instance Manager pods (one per data engine version), each with its own CPU reservation. During a Longhorn upgrade, a new Instance Manager pod is created alongside the existing one on each affected node. The old pod keeps hosting the engine and replica instances that are already running so that live volumes stay online, while newly created and newly attached volumes land on the upgraded pod. Existing volumes only move to the new Instance Manager when they are detached and reattached (typically as part of the engine upgrade workflow), and the old pod is only removed once no instances remain inside it. Because both the old and new pods keep their CPU and memory reservations during this window, each node needs enough spare capacity to run the extra pods until the upgrade completes.
+
+Because the engine and replica instances live inside the Instance Manager pod, the Instance Manager defines the failure domain for everything it hosts. If an Instance Manager pod is restarted or evicted, every engine and replica instance it was hosting goes with it, and the Longhorn Manager must reattach the affected volumes and rebuild replicas as needed. For this reason, Longhorn reserves CPU for the Instance Manager pod (see [Guaranteed Instance Manager CPU](../best-practices/#guaranteed-instance-manager-cpu)) and avoids restarting it while it is still hosting active instances. For the same reason, Longhorn protects the Instance Manager pod from accidental eviction or drain with a PodDisruptionBudget, whose behavior is controlled by the [Node Drain Policy](../references/settings/#node-drain-policy) setting.
+
+This also means that the Instance Manager is the pod you target when debugging engine or replica behavior — there is no separate per-volume engine pod. Instance Manager pods follow the naming pattern `instance-manager-<hash>` and carry the `longhorn.io/node=<node-name>` label, which is the practical entry point for `kubectl logs` and `kubectl exec` when troubleshooting volume I/O issues on a specific node.
+
+## 1.3. Advantages of a Microservices-Based Design
 
 In Longhorn, each Engine only needs to serve one volume, simplifying the design of the storage controllers. Because the failure domain of the controller software is isolated to individual volumes, a controller crash will only impact one volume.
 
@@ -83,14 +105,14 @@ Because each volume has its own controller, the controller and replica instances
 
 Longhorn can create a long-running job to orchestrate the upgrade of all live volumes without disrupting the on-going operation of the system. To ensure that an upgrade does not cause unforeseen issues, Longhorn can choose to upgrade a small subset of the volumes and roll back to the old version if something goes wrong during the upgrade.
 
-## 1.3. CSI Driver
+## 1.4. CSI Driver
 
 The Longhorn CSI driver takes the block device, formats it, and mounts it on the node. Then the [kubelet](https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet/) bind-mounts the device inside a Kubernetes Pod. This allows the Pod to access the Longhorn volume.
 
 The required Kubernetes CSI Driver images will be deployed automatically by the longhorn driver deployer.
 To install Longhorn in an air gapped environment, refer to [this section](../deploy/install/airgap).
 
-## 1.4. CSI Plugin
+## 1.5. CSI Plugin
 
 Longhorn is managed in Kubernetes via a [CSI Plugin.](https://kubernetes-csi.github.io/docs/) This allows for easy installation of the Longhorn plugin.
 
@@ -106,7 +128,7 @@ In contrast, v2 volumes come with different prerequisites, depending on the conf
 - For the NVMe-TCP frontend, the `nvme_tcp` module is necessary.
 - For the UBLK frontend, both the `ublk_drv` module and huge pages support must be enabled.
 
-## 1.5. The Longhorn UI
+## 1.6. The Longhorn UI
 
 The Longhorn UI interacts with the Longhorn Manager through the Longhorn API, and acts as a complement of Kubernetes. Through the Longhorn UI, you can manage snapshots, backups, nodes and disks.