docs(concepts): add Instance Manager section #1381
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -15,10 +15,11 @@ For the installation requirements, go to [this section.](../deploy/install/#inst | |
|
|
||
| - [1. Design](#1-design) | ||
| - [1.1. The Longhorn Manager and the Longhorn Engine](#11-the-longhorn-manager-and-the-longhorn-engine) | ||
| - [1.2. The Instance Manager](#12-the-instance-manager) | ||
| - [1.3. Advantages of a Microservices Based Design](#13-advantages-of-a-microservices-based-design) | ||
|
Felipalds marked this conversation as resolved.
Outdated
|
||
| - [1.4. CSI Driver](#14-csi-driver) | ||
| - [1.5. CSI Plugin](#15-csi-plugin) | ||
| - [1.6. The Longhorn UI](#16-the-longhorn-ui) | ||
| - [2. Longhorn Volumes and Primary Storage](#2-longhorn-volumes-and-primary-storage) | ||
| - [2.1. Thin Provisioning and Volume Size](#21-thin-provisioning-and-volume-size) | ||
| - [2.2. Reverting Volumes in Maintenance Mode](#22-reverting-volumes-in-maintenance-mode) | ||
|
|
@@ -73,7 +74,24 @@ In the figure below, | |
|
|
||
| {{< figure alt="read/write data flow between the volume, controller instance, replica instances, and disks" src="/img/diagrams/architecture/how-longhorn-works-with-kubernetes.svg" >}} | ||
|
|
||
| ## 1.2. The Instance Manager | ||
|
|
||
| The [Instance Manager](https://github.com/longhorn/longhorn-instance-manager) is the per-node component that hosts and manages the lifecycle of engine and replica instances. It runs as a pod in the `longhorn-system` namespace, and is created and supervised by the Longhorn Manager. Unlike the Longhorn Manager, which is a single DaemonSet across the cluster, the Instance Manager is a system-managed component whose lifecycle is owned by Longhorn itself. | ||
|
|
||
| When the Longhorn Manager decides to create or attach a volume, it does not start the engine or replica processes directly. Instead, it instructs the Instance Manager on the relevant node to start them inside the Instance Manager pod. The same Instance Manager hosts the engine and replica instances for many volumes that happen to land on its node. For a given volume, one engine instance lives in the Instance Manager on the node where the workload Pod runs, and one replica instance lives in the Instance Manager on each node selected for that volume's replicas. The number of replicas per volume is controlled by the [Default Replica Count](../references/settings/#default-replica-count) setting and can be overridden per volume. | ||
|
shuo-wu marked this conversation as resolved.
Outdated
|
||
|
|
||
| The hosting model differs between data engines: | ||
|
|
||
| - **V1 Data Engine.** The Instance Manager runs each engine and each replica as a Linux process inside the pod. The engine process is also what exposes the volume's block device to the host, using iSCSI as the frontend. A single V1 Instance Manager pod can host engine and replica processes for many volumes. | ||
| - **V2 Data Engine.** The Instance Manager runs an SPDK target process (`spdk_tgt`) inside the pod. Engines are exposed as SPDK RAID block devices and replicas as SPDK logical volume bdevs within that target. The frontend presenting the block device to the host (NVMe-TCP or UBLK) is also driven from this Instance Manager. Because `spdk_tgt` uses busy-polling, the V2 Instance Manager consumes at least one dedicated CPU core per node. | ||
|
shuo-wu marked this conversation as resolved.
Outdated
|
||
|
|
||
| If both data engines are enabled on a cluster, each node runs two Instance Manager pods — one for V1 and one for V2 — each with its own CPU reservation. | ||
|
Contributor
There was a problem hiding this comment. It's for one version. I think you can talk a little bit more about the upgrade case in this paragraph. |
||
|
|
||
| Because the engine and replica instances live inside the Instance Manager pod, the Instance Manager defines the failure domain for everything it hosts. If an Instance Manager pod is restarted or evicted, every engine and replica instance it was hosting goes with it, and the Longhorn Manager must reattach the affected volumes and rebuild replicas as needed. For this reason, Longhorn reserves CPU for the Instance Manager pod (see [Guaranteed Instance Manager CPU](../best-practices/#guaranteed-instance-manager-cpu)) and avoids restarting it while it is still hosting active instances. | ||
|
Contributor
There was a problem hiding this comment. Since an instance manager pod missing means all replicas on this node are unavailable, Longhorn will avoid the volume data loss by relying on PDB to protect the instance manager pods from accidental eviction or drain, see https://longhorn.io/docs/1.12.0/references/settings/#node-drain-policy |
||
|
|
||
| This also means that the Instance Manager is the pod you target when debugging engine or replica behavior — there is no separate per-volume engine pod. Instance Manager pods follow the naming pattern `instance-manager-<hash>` and carry the `longhorn.io/node=<node-name>` label, which is the practical entry point for `kubectl logs` and `kubectl exec` when troubleshooting volume I/O issues on a specific node. | ||
|
|
||
| ## 1.3. Advantages of a Microservices Based Design | ||
|
Felipalds marked this conversation as resolved.
Outdated
|
||
|
|
||
| In Longhorn, each Engine only needs to serve one volume, simplifying the design of the storage controllers. Because the failure domain of the controller software is isolated to individual volumes, a controller crash will only impact one volume. | ||
|
|
||
|
|
@@ -83,14 +101,14 @@ Because each volume has its own controller, the controller and replica instances | |
|
|
||
| Longhorn can create a long-running job to orchestrate the upgrade of all live volumes without disrupting the on-going operation of the system. To ensure that an upgrade does not cause unforeseen issues, Longhorn can choose to upgrade a small subset of the volumes and roll back to the old version if something goes wrong during the upgrade. | ||
|
|
||
| ## 1.4. CSI Driver | ||
|
|
||
| The Longhorn CSI driver takes the block device, formats it, and mounts it on the node. Then the [kubelet](https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet/) bind-mounts the device inside a Kubernetes Pod. This allows the Pod to access the Longhorn volume. | ||
|
|
||
| The required Kubernetes CSI Driver images will be deployed automatically by the longhorn driver deployer. | ||
| To install Longhorn in an air gapped environment, refer to [this section](../deploy/install/airgap). | ||
|
|
||
| ## 1.5. CSI Plugin | ||
|
|
||
| Longhorn is managed in Kubernetes via a [CSI Plugin.](https://kubernetes-csi.github.io/docs/) This allows for easy installation of the Longhorn plugin. | ||
|
|
||
|
|
@@ -106,7 +124,7 @@ In contrast, v2 volumes come with different prerequisites, depending on the conf | |
| - For the NVMe-TCP frontend, the `nvme_tcp` module is necessary. | ||
| - For the UBLK frontend, both the `ublk_drv` module and huge pages support must be enabled. | ||
|
|
||
| ## 1.6. The Longhorn UI | ||
|
|
||
| The Longhorn UI interacts with the Longhorn Manager through the Longhorn API, and acts as a complement of Kubernetes. Through the Longhorn UI, you can manage snapshots, backups, nodes and disks. | ||
|
|
||
|
|
||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.