drivers/nvme: make interrupt coalescing opt-in, default off for low latency - #1481
Open
gburd wants to merge 1 commit into
Open
drivers/nvme: make interrupt coalescing opt-in, default off for low latency#1481gburd wants to merge 1 commit into
gburd wants to merge 1 commit into
Conversation
…atency The driver unconditionally programmed NVMe interrupt coalescing with threshold=20, time=2 (a 200us completion-aggregation window) on every non-QEMU controller. Coalescing reduces interrupt load for deep, streaming IO, but it adds latency to low-queue-depth synchronous IO: a request that issues a single command and waits for its completion (a database fsync, a filesystem journal or ZFS ZIL commit, and similar barrier operations) pays the full aggregation-timer delay on every such operation, which caps synchronous throughput. Default to no coalescing for the lowest completion latency, and let workloads that prefer fewer interrupts opt back in with OSV_NVME_INT_COALESCING="threshold,time100us" (threshold in completions, time in 100us units, matching the NVMe Set Features encoding). The real-controller guard is preserved since QEMU's emulated controller ignores the feature.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The NVMe driver unconditionally programmed interrupt coalescing (
threshold=20,time=2, a 200us completion-aggregation window) on every non-QEMU controller. Coalescing reduces interrupt load for deep, streaming I/O, but it adds latency to low-queue-depth synchronous I/O: a request that issues a single command and waits for its completion pays the full aggregation-timer delay on every such operation.This hurts barrier-style operations that dominate database and journaling workloads (an fsync, a filesystem journal commit, a ZFS ZIL commit), where each commit issues one FLUSH and waits for it. With the 200us window, every commit is delayed up to 200us before its completion interrupt fires, capping synchronous write throughput.
Change
OSV_NVME_INT_COALESCING="threshold,time100us"for workloads that prefer fewer interrupts (threshold in completions, time in 100us units, matching the NVMe Set Features encoding).Measured effect
On real NVMe, disabling coalescing measurably improves synchronous-commit write throughput at low-to-moderate concurrency. There is a genuine tradeoff: streaming read throughput benefits from coalescing, so the env knob lets those workloads restore it. Defaulting off favors latency-sensitive workloads, which is the more common surprise when moving from an emulated controller (no coalescing) to real hardware.