virtio-net: implement multiqueue (VIRTIO_NET_F_MQ) - #1463
Conversation
Update: benchmark results and a regression to fix before mergeI ran a direct A/B (multiqueue nqp>=8 vs single-queue nqp=1, same image, same NIC, an external client) with PostgreSQL/pgbench across concurrency levels, and the results change what I can claim for this PR. Converting to draft. Two honest findings: 1. The multiqueue feature does not improve concurrent throughput on this workload. Single-queue (nqp=1) equals or beats nqp>=8 at every concurrency level (e.g. read-only tps at 16 connections: nqp=1 gave ~66k vs nqp=16 ~25k, reproduced). The throughput ceiling this workload hits is a per-connection CPU/scheduler cost and a storage write path, not RX-queue depth. So multiqueue is not the scaling win I expected; it is correct and it activates (N queues, N Rx threads, device ACKs CTRL_MQ_VQ_PAIRS_SET), but it does not move the numbers here. 2. The MSI-X vector budget introduces a virtio-blk regression under load. When virtio-net claims 2*nqp vectors at high vCPU count, it starves the several virtio-blk devices; the soft-fail path (register_interrupt_handler returning 0 instead of aborting) then leaves a blk queue with no interrupt vector, so completions are never delivered and reads fail with EIO under sustained write. The pre-MQ image was clean. This needs fixing: a queue that cannot get its own vector must fall back to a shared/polled vector rather than run interrupt-less, and net must not be allowed to starve blk. What stands: the vector-budget change does fix a real pre-existing boot crash (many-vCPU guests with several multiqueue virtio-blk devices previously aborted in register_interrupt_handler on IDT vector exhaustion; they now boot). That part is worth keeping. Plan: I will either (a) fix the vector arbitration so blk always keeps a working (shared/polled) vector and re-take the A/B, or (b) split the boot-crash MSI-X fix into its own PR and hold the multiqueue feature until it demonstrates a real gain. Marking draft until then. |
|
I am not surprised by your performance findings. I am seeing similar ones with the virtio-blk multi-queue multi-consumer -threads changes. Single queue, single consumer thread seems to be beating all other setups. |
|
That lines up with what I found, and it points at a shared root cause rather than anything queue-specific. I profiled the concurrency collapse separately (a many-connection server workload over the network) and the guest was ~90% idle while throughput fell: threads were piling onto a few CPUs while the rest sat idle. The scheduler places a new thread on its creator's CPU and re-wakes a blocked thread on its last CPU, and the load balancer only migrated one thread per 100ms, far too slow to disperse a fan-out of many worker threads. That is why more queues (net or blk) did not help: adding consumers does not change where the woken threads land, so they still serialize on the same few CPUs. I opened #1464 for the scheduler side (run the balancer more often and drain the whole imbalance per pass); it recovered +53% to +73% at 32 to 64 connections in an A/B. Your virtio-blk observation is consistent with the same effect: single queue/single consumer wins because it avoids paying migration/placement overhead for consumers that the balancer cannot spread anyway. Given all that, I am holding this virtio-net MQ PR as draft. The multiqueue feature does not earn its keep on its own, and the vector-budget change in it also needs the virtio-blk starvation fix I noted above. The genuinely useful part is the MSI-X vector-budget boot fix (many-vCPU guests with several multiqueue virtio-blk devices previously aborted at boot); if useful I can split that out into its own small PR and drop the multiqueue machinery. |
3ed80cc to
00f2b61
Compare
|
Reworked: the MSI-X vector-budget logic this originally carried is factored out into #1465 (so blk can never be starved of interrupt vectors), and this PR now stacks on it with the multiqueue mechanism alone (N Rx virtqueues + N receiver threads pinned across CPUs, N Tx queues selected by CPU, CTRL_MQ_VQ_PAIRS_SET). Honest performance note: on a many-connection PostgreSQL/pgbench workload I measured MQ vs single-queue on an otherwise-fixed multi-CPU guest and saw no reliable throughput change (gains within run-to-run noise; that workload is bottlenecked on wake/round-trip latency, not RX-queue depth, so more queues don't help it). So I am not claiming a benchmark win here. The case for it is architectural, not workload-specific: single-queue serializes all inbound traffic through one receiver thread on one CPU, which caps any RX-bound concurrent workload regardless of core count. Multiqueue removes that serialization for the workloads where RX genuinely is the bottleneck. I am fine keeping it as draft pending an RX-bound benchmark that demonstrates the gain, or landing it as a general capability with the MSI-X budget (#1465) that makes it safe. Reviewers' call on that basis. |
The x86-64 IDT exposes only 224 usable interrupt vectors (32..255), and
every device shares that pool. A guest with many vCPUs and several
multiqueue virtio devices can request one MSI-X vector per queue and
exhaust the pool during boot. Two things then went wrong:
- register_interrupt_handler() called abort() when no IDT vector was
free, crashing the boot instead of letting the device degrade.
- request_vectors() pushed a msix_vector whose allocation had failed,
and ~msix_vector() then unregistered vector 0.
Fix both and add a soft cap so devices stay within budget in the first
place:
- register_interrupt_handler() returns 0 (an impossible real vector,
since 0..31 are CPU exceptions) as an "exhausted" sentinel instead
of aborting. msix_vector's destructor skips the unregister when its
vector is 0, and request_vectors() stops at the first failed
allocation and returns the short vector it did get; the existing
easy_register() already treats a short result as failure, so a
driver falls back to fewer queues rather than crashing.
- Add virtio_driver::reserve_msix_vectors(): a process-wide soft
budget that a multiqueue driver consults to cap how many queues it
arms with a distinct interrupt, leaving headroom for other devices
and always granting at least one. virtio-blk uses it to cap its
active queue count; extra probed virtqueues are simply left unused.
Signed-off-by: Greg Burd <greg@burd.me>
The virtio-net driver hardcoded a single Rx/Tx queue pair drained by one
receive poll thread. Every inbound packet for every connection funnelled
through that one queue and one thread on one CPU, so receive-side network
throughput could not scale past what a single core can drain, regardless of
how many vCPUs the guest had.
Negotiate VIRTIO_NET_F_MQ (and VIRTIO_NET_F_CTRL_VQ, required to activate it)
and drive N Rx/Tx queue pairs:
- Read max_virtqueue_pairs from config space and pick the pair count as
min(advertised pairs, number of vCPUs, MSI-X vector budget).
- Replace the single rxq/txq with vectors of queue pairs. Each Rx queue has
its own poll thread, pinned to a distinct CPU so receivers run in parallel;
each Tx queue has its own xmitter. Transmit selects a Tx queue by CPU id.
- Register one MSI-X vector per data virtqueue (entry = virtqueue index, per
the 1:1 mapping setup_queue() programs) so a completion on queue i wakes
that queue's thread.
- Send VIRTIO_NET_CTRL_MQ_VQ_PAIRS_SET on the control virtqueue to tell the
device how many pairs the guest will use.
When the device does not offer VIRTIO_NET_F_MQ (or offers a single pair, as
with the MMIO transport's shared interrupt) the driver uses exactly one queue
pair and behaves as before.
The per-queue MSI-X vectors are drawn from the shared vector budget added in
the preceding change (virtio_driver::reserve_msix_vectors), so a high-vCPU
guest with this multiqueue net device plus other multiqueue virtio devices
stays within the 224 usable IDT vectors and each device still gets at least
one interrupt-bearing queue instead of exhausting the pool.
Signed-off-by: Greg Burd <greg@burd.me>
00f2b61 to
1454fd4
Compare
Follow-up: an RX-bound benchmark now demonstrates the gainEarlier I held this draft because an isolated MQ-vs-single-queue A/B showed no reliable throughput change: that workload was bottlenecked on wake/dispatch latency downstream of RX, so more RX queues could not help it, and the effect of multiqueue was masked. With that downstream bottleneck fixed as well (the idle-CPU pull dispatch work in #1472), the picture separates cleanly and multiqueue earns its keep. On a workload with many concurrent network connections driven over a real NIC from an external client:
So the architectural claim I made before is now backed by a measured RX-bound result: N concurrent connections spread across N CPUs instead of serializing through a single receiver. Multiqueue is necessary but not sufficient on its own; paired with the dispatch fix it is a real, repeatable win. Dependency: this still stacks on the MSI-X vector-budget fix (#1465, so many-queue guests keep working interrupt vectors for every device); #1465 should land first. Marking ready for review on that basis. |
2 participants
What
OSv's virtio-net driver hardcodes a single Rx/Tx virtqueue pair drained by one
receiver()poll thread. Every guest's inbound network I/O funnels through that one queue on one CPU.VIRTIO_NET_F_MQexists as a feature constant but was never negotiated.This implements multiqueue:
VIRTIO_NET_F_MQ+VIRTIO_NET_F_CTRL_VQ; readmax_virtqueue_pairsfrom device config; usenqp = min(advertised, num_vCPUs, MSI-X budget)._rxq/_txqsingle structs become N-element vectors; N Rx poll threads, one per pair, pinned across distinct CPUs.VIRTIO_NET_CTRL_MQ_VQ_PAIRS_SETissued on the control queue.nqp == 1path (device without MQ, e.g. QEMU SLIRP) is unchanged.Prerequisite fix: MSI-X vector budget (also fixes a boot crash)
Enabling multiqueue on a many-vCPU guest exposed a pre-existing crash: OSv
abort()s ininterrupt_descriptor_table::register_interrupt_handlerwhen the 224 usable IDT vectors (32..255) are exhausted. With several multiqueue virtio-blk devices at high-smpthis already crashed at boot before any net change. This PR adds a process-wide MSI-X vector budget:register_interrupt_handlerreturns 0 (no vector) instead of aborting when the IDT is full, and virtio-blk and virtio-net cap their interrupt-bearing queues against the shared pool.Validation
CTRL_MQ_VQ_PAIRS_SET(8)reaching DRIVER_OK. A single-queue device falls back to nqp=1.-smp 4,-smp 16, and-smp 48.-smp 48): stock master aborts 3/3 inregister_interrupt_handler; patched boots 5/5.Note on throughput
The motivation is concurrent-workload scaling (single-queue net serializes many concurrent connections' RX through one thread). I verified the MQ mechanism (8 active queues/threads) and boot stability here; the end-to-end concurrent-throughput curve is best shown with a many-connection server workload and I have not included those numbers in this PR. Bulk single-stream TCP is not a faithful measure (vhost GRO coalesces it so one Rx thread suffices). The change is a strict superset of the current single-queue behavior (nqp=1 unchanged), so it is safe independent of the throughput delta.