docs(concepts): add Instance Manager section - #1381
Open
Felipalds wants to merge 3 commits into
Open
Conversation
✅ Deploy Preview for longhornio ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
Felipalds
force-pushed
the
docs/concepts-instance-manager
branch
from
June 17, 2026 20:39
7ece590 to
b6263b3
Compare
Signed-off-by: Luiz Felipe Rosa <luizrosa8@proton.me>
Felipalds
force-pushed
the
docs/concepts-instance-manager
branch
from
June 17, 2026 20:40
b6263b3 to
5e16d3c
Compare
sushant-suse
previously approved these changes
Jun 18, 2026
Felipalds
force-pushed
the
docs/concepts-instance-manager
branch
from
June 19, 2026 00:32
197c439 to
bf26589
Compare
Co-authored-by: Sushant Gaurav <sushant.gaurav@suse.com> Signed-off-by: Luiz Rosa <luiz.rosa@suse.com>
Felipalds
force-pushed
the
docs/concepts-instance-manager
branch
from
June 19, 2026 00:34
bf26589 to
cef710b
Compare
Author
|
Thanks for the review @sushant-suse ! |
sushant-suse
previously approved these changes
Jun 19, 2026
shuo-wu
reviewed
Jun 26, 2026
shuo-wu
left a comment
shuo-wu
left a comment
Contributor
There was a problem hiding this comment.
Left some supplementary comments for the instance manager section. Feel free to reorganize the words.
| - **V1 Data Engine.** The Instance Manager runs each engine and each replica as a Linux process inside the pod. The engine process is also what exposes the volume's block device to the host, using iSCSI as the frontend. A single V1 Instance Manager pod can host engine and replica processes for many volumes. | ||
| - **V2 Data Engine.** The Instance Manager runs an SPDK target process (`spdk_tgt`) inside the pod. Engines are exposed as SPDK RAID block devices and replicas as SPDK logical volume bdevs within that target. The frontend presenting the block device to the host (NVMe-TCP or UBLK) is also driven from this Instance Manager. Because `spdk_tgt` uses busy-polling, the V2 Instance Manager consumes at least one dedicated CPU core per node. | ||
|
|
||
| If both data engines are enabled on a cluster, each node runs two Instance Manager pods — one for V1 and one for V2 — each with its own CPU reservation. |
Contributor
There was a problem hiding this comment.
It's for one version. I think you can talk a little bit more about the upgrade case in this paragraph.
|
|
||
| If both data engines are enabled on a cluster, each node runs two Instance Manager pods — one for V1 and one for V2 — each with its own CPU reservation. | ||
|
|
||
| Because the engine and replica instances live inside the Instance Manager pod, the Instance Manager defines the failure domain for everything it hosts. If an Instance Manager pod is restarted or evicted, every engine and replica instance it was hosting goes with it, and the Longhorn Manager must reattach the affected volumes and rebuild replicas as needed. For this reason, Longhorn reserves CPU for the Instance Manager pod (see [Guaranteed Instance Manager CPU](../best-practices/#guaranteed-instance-manager-cpu)) and avoids restarting it while it is still hosting active instances. |
Contributor
There was a problem hiding this comment.
Since an instance manager pod missing means all replicas on this node are unavailable, Longhorn will avoid the volume data loss by relying on PDB to protect the instance manager pods from accidental eviction or drain, see https://longhorn.io/docs/1.12.0/references/settings/#node-drain-policy
Member
|
@Felipalds Could you address the comments from @shuo-wu? Thanks. |
Signed-off-by: Luiz Rosa <luiz.rosa@suse.com>
Felipalds
force-pushed
the
docs/concepts-instance-manager
branch
from
July 6, 2026 21:33
961cc0b to
79b1efa
Compare
Author
|
Thanks @shuo-wu. I addressed all comments. Would like to see if they are ok, specially on the upgrade part. Thanks. |
Author
shuo-wu
approved these changes
Aug 12, 2026
|
|
||
| If both data engines are enabled on a cluster, each node runs two Instance Manager pods (one per data engine version), each with its own CPU reservation. During a Longhorn upgrade, a new Instance Manager pod is created alongside the existing one on each affected node. The old pod keeps hosting the engine and replica instances that are already running so that live volumes stay online, while newly created and newly attached volumes land on the upgraded pod. Existing volumes only move to the new Instance Manager when they are detached and reattached (typically as part of the engine upgrade workflow), and the old pod is only removed once no instances remain inside it. Because both the old and new pods keep their CPU and memory reservations during this window, each node needs enough spare capacity to run the extra pods until the upgrade completes. | ||
|
|
||
| Because the engine and replica instances live inside the Instance Manager pod, the Instance Manager defines the failure domain for everything it hosts. If an Instance Manager pod is restarted or evicted, every engine and replica instance it was hosting goes with it, and the Longhorn Manager must reattach the affected volumes and rebuild replicas as needed. For this reason, Longhorn reserves CPU for the Instance Manager pod (see [Guaranteed Instance Manager CPU](../best-practices/#guaranteed-instance-manager-cpu)) and avoids restarting it while it is still hosting active instances. For the same reason, Longhorn protects the Instance Manager pod from accidental eviction or drain with a PodDisruptionBudget, whose behavior is controlled by the [Node Drain Policy](../references/settings/#node-drain-policy) setting. |
Contributor
There was a problem hiding this comment.
NIT: Besides, Longhorn deploys a dedicated Priority Class (see setting Priority Class) by default to help prevent Longhorn components from being evicted under Node Pressure.
| - **V1 Data Engine.** The Instance Manager runs each engine and each replica as a Linux process inside the pod. The engine process is also what exposes the volume's block device to the host, using iSCSI as the frontend. A single V1 Instance Manager pod can host engine and replica processes for many volumes. Because engine and replica processes share the pod, the Instance Manager's resource consumption scales with the aggregate I/O load of the volumes hosted on the node. Review the [Guaranteed Instance Manager CPU](../references/settings/#guaranteed-instance-manager-cpu) setting before scaling replicas or attaching high-throughput volumes on a node. | ||
| - **V2 Data Engine.** The Instance Manager runs an SPDK target process (`spdk_tgt`) inside the pod, and SPDK takes over the full storage path, including the disks themselves. Each V2 block-type disk is imported into the target as a Logical Volume Store (LVS), and replicas live on top as SPDK logical volume bdevs. Engines are exposed as SPDK RAID block devices built from those replicas. The frontend presenting the block device to the host (NVMe-TCP or UBLK) is also driven from this Instance Manager. Because `spdk_tgt` runs in polling mode by default, the V2 Instance Manager reserves dedicated CPU cores and memory (hugepages when enabled) on each node. Use the [Data Engine CPU Mask](../references/settings/#data-engine-cpu-mask) and [Data Engine Memory Size](../references/settings/#data-engine-memory-size) settings to tune these reservations. Longhorn also supports [Interrupt Mode](../advanced-resources/v2-data-engine/interrupt-mode/) as an alternative when reducing CPU consumption is more important than raw I/O performance. | ||
|
|
||
| If both data engines are enabled on a cluster, each node runs two Instance Manager pods (one per data engine version), each with its own CPU reservation. During a Longhorn upgrade, a new Instance Manager pod is created alongside the existing one on each affected node. The old pod keeps hosting the engine and replica instances that are already running so that live volumes stay online, while newly created and newly attached volumes land on the upgraded pod. Existing volumes only move to the new Instance Manager when they are detached and reattached (typically as part of the engine upgrade workflow), and the old pod is only removed once no instances remain inside it. Because both the old and new pods keep their CPU and memory reservations during this window, each node needs enough spare capacity to run the extra pods until the upgrade completes. |
Contributor
There was a problem hiding this comment.
If there are no enough resources reserved, Longhorn will not start the new version IM pod after the system upgrade, especially for v2 IM (which requires hugepages and dedicated CPU masks).
Member
|
@Felipalds Could you address the comments? Thanks. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The Instance Manager is currently only mentioned briefly in
terminology.mdandcontributing.md, so readers ofconcepts.mdare not introduced to the per-node pod that actually hosts engine and replica processes. This adds a new1.2. The Instance Managersection toconcepts.mdcovering:Applied to the supported 1.11.x, 1.12.x, and 1.13.0 docs. Renumbered the existing 1.2 to 1.5 subsections to 1.3 to 1.6 (headings + TOC).
Any feedback on this PR is welcome, please suggest additions, removals, or rewordings if anything is inaccurate or missing.
Issue related: longhorn/longhorn#13197