sched: idle-CPU pull work-stealing to disperse woken threads under concurrency

[gburd]

Summary

Adds an idle-CPU pull (work-stealing) path to the scheduler so that when many
threads are woken in a tight wake/run/block cycle by a single waker, the work is
dispersed across idle CPUs immediately instead of piling onto the waker's home
CPU.

The problem

When one thread wakes many workers in quick succession -- for example a single
network RX thread waking a worker per request -- the woken threads tend to run on
the waker's home CPU while other CPUs sit idle. The existing load_balance()
only rebalances on its 100ms timer, which is far too slow for a cycle that never
accumulates a visible runqueue: each thread runs briefly and blocks again before
the balancer's next tick, so the imbalance is never observed and the machine
stays lopsided with many idle CPUs.

The mechanism

Add a pull step on the idle path (do_idle()): when a CPU is about to poll/halt
with an empty runqueue, it scans for the busiest CPU and, via an IPI the busiest
CPU services in its own interrupt context, has that CPU donate one migratable
runnable thread. All runqueue and per-CPU timer mutation happens on the owning
(victim) CPU, so it is same-CPU-safe -- byte-identical bookkeeping to
load_balance()'s source-side migration.

The key design choice is to pull (idle CPU asks) rather than push at wake
time (waker steers each woken thread). Pulling is self-throttling by
construction:

• only idle CPUs ask, and the moment a CPU receives donated work it is no longer
  idle and stops asking;
• a single per-victim request slot (pull_donate_to) bounds it to one
  outstanding request per victim.

So the IPI rate is bounded by the number of idle CPUs, never by the wake rate,
and cannot build into a per-wake IPI storm. It runs on every idle transition,
which is far more responsive than the 100ms load balancer for wake/run/block
workloads.

Gating / safety

Gated by a new CONF_sched_wake_pull (default y) and additionally a runtime
no-op unless the OSV_WAKE_PULL=1 environment toggle is set at boot, so the
default build behaves exactly as before. This makes it easy to A/B a single
image: OSV_WAKE_PULL=0 (stock behavior) vs OSV_WAKE_PULL=1 (pull enabled).

Motivating measurement

PostgreSQL 18 read-only pgbench, single-queue virtio-net, matched OSv-vs-Linux
KVM guests on one host, every cell 0-failed, best of rounds. OSv throughput with
the toggle off vs on:

clients	OSv off (tps)	OSv on (tps)	change	OSv/Linux off -> on
c16	36890	116175	+215%	0.23x -> 0.71x
c32	30193	91412	+203%	0.17x -> 0.51x
c48	35471	86936	+145%

The mid-range concurrency cliff -- where idle CPUs previously went unused -- is
largely closed, and the full c1-c64 read-only and read-write sweeps complete with
no livelock. PostgreSQL is only the motivating workload here; this is a generic
scheduler improvement that benefits any concurrent workload with a single-waker
fan-out.

Notes

• Fork-independent: builds and links clean with conf_fork=0 (verified: fresh
  loader.elf links rc=0 on this branch off master).
• Diff is additive only (4 files, ~180 lines): core/sched.cc,
  include/osv/interrupt.hh, include/osv/sched.hh, conf/kconfig/threads.

[gburd]

Follow-up: benchmark validates this as the dispatch lever, plus a fork-only safety follow-on

A many-connection network workload (driven over a real NIC from an external client) confirmed this idle-CPU pull work-stealing is the fix for the dispatch collapse I described above. Without it, woken worker threads pile onto their last-run CPUs while other CPUs idle-spin, and throughput falls under rising concurrency on an otherwise-idle guest. With it, work spreads across the available CPUs (e.g. 14 of 16 busy where only 2 were before), throughput climbs with concurrency instead of collapsing, and more vCPUs finally help. It is the wake-to-run dispatch lever the concurrent workload needed, and it composes with the multiqueue receive work (#1463/#1470): MQ removes the single-receiver RX funnel, this removes the wake-dispatch funnel.

One safety follow-on for the fork build only: on a CONF_fork build, donate_to_puller() can migrate a parked application thread, but unlike load_balance()'s source-side migration it did not first cpu::unlink_parked() the thread. A donated parked thread was then left linked on the source CPU's per-CPU timer list, and a later timer IRQ on that CPU could walk the migrated thread's on-stack timer node through the wrong address space and fault in non-preemptable context. This path is unreachable on a stock (conf_fork=0) build, and the isolated fork+timer regression test does not exercise it because it never migrates a parked thread via the idle-pull path.

The fix mirrors load_balance()'s existing #if CONF_fork cpu::unlink_parked(mig) exactly and is conf_fork=0 byte-identical. Since it references machinery that only exists on the fork build, it is not standalone against master; it belongs on the fork stack (#1458) as a follow-on to this PR. Recorded here so the dependency is visible: this PR is correct as-is for a stock build, and the fork build needs the additional unlink.

[gburd]

Additional real-workload validation: PostgreSQL 18.6 on OpenZFS (HammerDB/pgbench TPC-B), m5d-class bare-metal KVM guest, 32 vCPUs, paired reboot-between-arms A/B (OSV_WAKE_PULL unset vs =1), pgbench scale 100, medians of 3-4 reps each.

pgbench clients	baseline TPS	OSV_WAKE_PULL=1 TPS	delta
8	7102	6674	-6%
16	11571	11340	-2%
32 (= vCPU count)	17090	17487	+2% (noise)
64	7819	22722	+191%
96	2166	3467	+60%

The oversubscribed regime (clients > vCPUs) is exactly the pattern the PR targets, and the mechanism is confirmed by direct instrumentation at 64 clients: without the pull the guest collapses to ~5-11 busy vCPUs (~8 host cores) while ~26 sit halted, because the single ZIL/commit waker's fan-out never accumulates a visible runqueue for the 100ms balancer to notice; with OSV_WAKE_PULL=1 idle vCPUs pull the woken backends immediately, reaching ~24-30 busy vCPUs (~2700% host CPU) and ~3x throughput, with p-latency dropping from 8.5ms to 2.9ms.

Honest scope note: at and below the vCPU count it is a wash-to-slightly-negative (at 1 client, -14%: with a single committer and idle CPUs the pull is pure overhead and cannot shorten the intrinsic cross-CPU wake-IPI/VM-exit path). So this is a clear win for oversubscribed concurrency and correctly a runtime-armed no-op by default. Boots clean; crash-marker grep clean across all runs.

[gburd]

Confirmed on a full database OLTP benchmark (TPC-C-like), in addition to the pgbench result above: PostgreSQL 18.6 on OpenZFS, bare-metal KVM guest, 32 vCPUs, 40-warehouse schema, timed run (1m rampup + 3m measure), paired A/B booted from a byte-identical clean pool snapshot (OSV_WAKE_PULL unset vs =1).

virtual users	baseline NOPM	OSV_WAKE_PULL=1 NOPM	delta
16	14149	15026	+6%
32 (= vCPU count)	26722	30087	+13%
64	7989	27009	+238%
96	7485	25697	+243%
128	7336	24821	+238%

Same mechanism, directly observed via per-vCPU thread state on the host: past the vCPU count the baseline collapses to ~8-10 busy vCPUs (the rest halted) as woken backends pile onto a few CPUs, while with the pull armed ~29-30 of 32 vCPUs stay busy and throughput tracks the utilization (the ~7.5k collapsed floor becomes a sustained ~25k). Boots clean, crash-marker grep clean.

Honest scope: this restores throughput/utilization under oversubscription; per-operation p99 latency is roughly flat across both arms (this workload is latency-bound on the storage record size), so the pull lifts throughput, not tail latency. Below the vCPU count it is a small win as expected. Confirms the feature does exactly what the PR describes, on a real workload.