[DRAFT / tracking - DO NOT MERGE as one unit] PostgreSQL-on-OSv fork+ZFS integration (will split into S1..S6 on #1455/#1456/#1457 + #1423) - #1458
Draft
gburd wants to merge 75 commits into
Conversation
…pt-in)
OSv is a single-address-space unikernel, so a literal copy-on-write fork() is
not the model. This adds a thread-backed fork() emulation that covers the
useful, compatible subset of fork semantics, gated entirely behind a new
CONFIG_fork kconfig option that defaults to OFF - so a default OSv build is
unchanged (none of this code is compiled in and fork()/vfork() return ENOSYS as
before).
With CONFIG_fork enabled:
- fork()/vfork() create a child OSv thread that resumes in fork()'s caller and
returns 0 in the child / the child pid in the parent (the classic twin
return). The child runs on a private copy of the parent's user stack, so
parent and child have independent locals after the return; it gets its own
fresh OSv per-thread TLS block (own errno etc).
Implemented in libc/process/fork.cc + arch/{x64,aarch64}/fork.cc.
- execve() launches the target as a fresh OSv application (its own ELF
namespace) and does not return, making fork()+exec() work.
- waitpid()/wait4()/wait() reap a child's exit status via a pid->child
registry; SIGCHLD is raised to the parent on child exit; SIGCHLD/SIGURG/
SIGWINCH now correctly default to ignore (not poweroff).
- exit()/_exit() in a fork child ends only that child, not the whole unikernel.
- pthread_atfork prepare/parent/child handlers are now actually run around
fork() (they were a no-op stub) - glibc/musl register these internally.
- sys_clone() routes the non-CLONE_THREAD (fork) case here when enabled.
Validated: tst-fork passes 10/10 on both x86-64 and aarch64 (twin return,
private-stack isolation, fork+exec, vfork, waitpid reaping).
Documented limitations (documentation/fork.md): the child shares the parent's
heap/globals (no per-process memory isolation in one address space - a follow-up
adds per-child copy-on-write address spaces behind the same flag); deep-call-
chain child unwind and fork-as-memory-snapshot (Redis BGSAVE) are not carried by
this base; execve()'s new-ELF-namespace path has a separate pre-existing fault.
Copyright (C) 2026 Greg Burd
execve() (CONF_fork) launches the target as a fresh OSv application in its own
ELF namespace via application::run(new_program=true) and, matching Linux, does
not return to the caller. A successful exec DID launch the program (the
elf::program new-namespace construction path is fine), but after the exec'd
program finished the unikernel would hang at shutdown instead of powering off:
the loader's application::join() blocked forever on the top-level app's
_terminated flag.
Root cause is the fork child thread's lifecycle, not execve or elf::program.
fork_thread() (arch/{x64,aarch64}/fork.cc) creates the child as a normal
*attached* sched::thread. At construction the thread captures a shared_ptr to
the current application's application_runtime (sched.cc sets _app_runtime =
app->runtime()). Nothing ever join()s the fork child -- the parent reaps it
through the fork pid registry / waitpid(), not sched::thread::join() -- so the
thread object is never destroyed and its _app_runtime shared_ptr is never
released. With that reference outstanding the application_runtime's use count
never reaches zero, ~application_runtime never runs, the app's _terminated is
never set, and application::join() waits forever. (This bit every fork(), and
was most visible after fork()+execve() where the child adopts the caller app's
runtime.)
Fix: create the fork child detached and dispose it in its cleanup.
- arch/{x64,aarch64}/fork.cc: mark the child attr().detached(), so on
completion it is handed to the thread reaper (which runs its set_cleanup()),
rather than sitting forever waiting to be joined.
- libc/process/fork.cc: the child's cleanup now also calls
sched::thread::dispose(child) after the existing bookkeeping (record exit
status if it fell off the end, free the copied user stack). Disposing the
thread releases its _app_runtime reference, letting the owning application's
runtime drop to zero so join() completes and OSv powers off. This mirrors
the default detached-thread cleanup ([this]{ dispose(this); }).
Both files are compiled only under CONF_fork, so default OSv builds are
unchanged.
Tests: tst-fork test 2 now execs a real payload (/tests/payload-exit7.so, which
prints a marker and exit(7)s) instead of the previous _exit(7) sentinel that
masked whether execve actually launched anything; a return from execve() is now
treated as a failure. Added tst-execve.so (fork+execve launches the payload and
its exit code is reaped; missing path returns -1/ENOENT) and payload-exit7.so.
Validated on x86-64 (KVM disk boot): tst-fork 10/10 and tst-execve 3/3 pass with
the real exec payload and OSv shuts down cleanly (5/5 repeat runs, no hang).
Copyright (C) 2026 Greg Burd
Follow-up to the base fork() PR: gives a forked child its own address space with copy-on-write of private mappings, so a forked child has real memory isolation like Linux fork() while MAP_SHARED / shm stays shared. Also gated behind CONFIG_fork (default off): with it disabled none of this is compiled and OSv's single-address-space model and context-switch path are unchanged. With CONFIG_fork enabled: - mmu::address_space object (page-table root + vma_list); the previous global becomes 'address space 0' (kernel + init app). - Per-thread current address space; the context switch loads the target CR3 only when the address space differs (a no-op for AS0-only workloads). Kernel PML4 entries are shared across all address spaces so OSv code + the kernel heap work identically after a switch. - fork() clones the parent's vmas into a child address space: PRIVATE writable mappings are write-protected in both and copied on the first write (reusing OSv's existing COW fault machinery); MAP_SHARED / shm map the same physical pages (truly shared). execve() returns the thread to AS0. - Lock/condvar wait_records for fork-child (non-AS0) threads are allocated from the AS-shared kernel heap instead of the thread stack, so a wait_record queued on a shared kernel mutex resolves to the same physical page for any waker across address spaces (kernel-stack coherence). Validated: tst-fork-cow proves a forked child's private memory stays private while MAP_SHARED stays shared; tst-fork stays 10/10. Known limitation (documented): the child's stack is still relocated+copied rather than same-VA COW, so a child that unwinds a very deep call chain (e.g. a multi-process server forking backends) can still hit a stack-fidelity issue (tst-fork-deep); the same-VA stack fix is a further follow-up. This PR delivers memory-isolated fork for the common cases with COW proven. Copyright (C) 2026 Greg Burd
Give a forked child its parent's EXACT user stack -- same virtual addresses, private physical pages -- instead of a relocated copy with a biased SP. A relocated+biased stack leaves every saved rbp, return address and &local in the copied frames pointing at the parent's stack (off by the bias), so a deep unwind in the child dereferences the parent and faults. tst-fork-deep (fork 12 frames deep, child unwinds all the way up and _exit()s a checked value) is the test that catches this; it now passes. Gated behind CONF_fork (default n); conf_fork=0 is unchanged and builds/boots clean (fork code fully excluded, tst-mmap passes). Four coupled changes make same-VA correct, layered on the Stage 2 COW address space and the fork-child lifecycle fix (detached child + dispose(child) in the reaper): 1. core/mmu.cc: clone_address_space() PRIVATIZES the forking thread's live stack VA range into the child AS -- fresh private pages that byte-copy the parent's stack, mapped writable (not COW: OSv runs kernel code with irqs off on the app stack, a COW fault there is illegal) at the SAME VA. The parent's PTEs are untouched. 2. arch/x64/fork.cc + libc/process/fork.cc + include/osv/fork.hh: the child resumes with the caller's FULL callee-saved register context (rbx, rbp, r12-r15, rsp, rip), captured at fork() entry into osv::fork_resume_ctx and restored by the child trampoline off a scratch base register (so the restore never clobbers its own base). fork() jmps past its own epilogue, so without this the caller would resume with fork()'s internal register values -- a deep chain that keeps an accumulator in a callee-saved reg across fork() computes garbage. 3. arch/x64/arch-switch.hh: defer the fork CR3 switch to EXACTLY the rsp/rbp swap inside switch_to()'s asm. Loading the incoming AS's CR3 early, while still executing on the outgoing thread's stack (the fpu-cw/mxcsr scratch saves), resolved those shared-VA stack writes through the wrong address space and clobbered a blocked thread's live stack (root-caused via hardware watchpoint: switch_to wrote garbage into a blocked parent's on-stack user_mutex slot in AS0). 4. core/lfmutex.cc + core/mmu.cc: a wait_record on a shared kernel condvar/mutex must live in the identity-mapped kernel heap (coherent in every AS) whenever a cross-address-space wake is possible. Extend the Option-A rule from 'current thread is a fork child' to also cover 'any child address space is live' (live_child_address_spaces counter), so the parent in AS0 -- woken by a child that walks the queue through its own page tables -- is safe too. libc/process/execve.cc: on a fork child, move the thread onto its own kernel stack before switching to AS0 and running the target, so reloading CR3 to AS0 cannot alias the same-VA app stack onto the parent's physical page (which would corrupt a blocked parent). The child address space is destroyed HERE, on the kernel stack, exactly once: the reap-time cleanup (libc/process/fork.cc) then sees the thread's AS is AS0 (!= child_as) and skips its own destroy, so destroy_address_space() runs once whether the child exits normally (reaper destroys) or execs (execve destroys) -- no leak, no double-free. A pre-exec existence check keeps a failed exec returning to the caller. Validated on x86-64 (KVM): tst-fork 10/10, tst-fork-cow 6/6, tst-fork-deep 0 failures, tst-execve 3/3 with clean shutdown (no hang, no double-free at reap), on smp=1 and smp=4, 5/5 repeat runs; conf_fork=0 builds and boots clean (fork excluded, tst-mmap passes).
…nZFS + patch series
Make the in-kernel ZFS implementation selectable at build time and integrate
OpenZFS 2.4.x without maintaining a fork, layered cleanly on the current
master (which already carries the reworked pagecache/ARC bridge, block
multiqueue, io_uring, musl 1.2.1, and modern-toolchain support).
conf_zfs switch (default bsd):
- conf_zfs=bsd - legacy in-tree BSD/Illumos ZFS (default, unchanged path)
- conf_zfs=openzfs - vendored OpenZFS 2.4.x from external/openzfs
Only one is compiled per build; both produce libsolaris.so plus the
zpool/zfs userspace. The shared solaris common objects (avl/nvpair/unicode/
fm/list) are provided by OpenZFS in openzfs mode and by the in-tree copies in
bsd mode.
Fork-free OpenZFS:
- external/openzfs pinned to the PUBLISHED upstream tag zfs-2.4.2
(github.com/openzfs/zfs @ 6330a45b), a plain submodule with no OSv fork.
- The OSv platform-layer changes live as a 19-patch git series in
modules/open_zfs/patches/ and are applied at build time by the Makefile
(idempotent via the external/openzfs/.osv-patches-applied stamp) only in
openzfs mode.
- .gitignore keeps an explicit !modules/open_zfs/patches/*.patch exception so
the series is tracked despite the global *.patch ignore.
ZFS-implementation-agnostic kernel:
- cv_timedwait keeps the BSD relative-timeout semantics; a distinct
openzfs_cv_timedwait (absolute deadline) is added in bsd/porting/netport1.cc
and exported, so the same loader.elf serves both. OpenZFS objects get
-Dcv_timedwait=openzfs_cv_timedwait via OPENZFS_CFLAGS in openzfs_sources.mk.
- core/pagecache.cc keeps master's live map_arc_buf/register_pagecache_arc_funs
bridge (used by conf_zfs=bsd) and additively provides the three symbols the
OpenZFS module needs (osv_free_pages, osv_pagecache_map_arc_page,
osv_pagecache_register_arc_rele) via a separate borrowed-ARC-page path into
read_cache. The borrow class is renamed cached_page_arc_borrow to avoid
colliding with master's arc_buf_t-based cached_page_arc.
- drivers/virtio-blk.cc: give the completion thread a 256 KB stack so the ZFS
vdev_disk_bio_done path does not overrun the default kernel stack.
Also: OpenZFS 2.4.x userspace lib/cmd build rules, openzfs_osv_compat.c, the
zfs-tools module.py (drops libzutil/libshare/libzfs_core/libtpool in bsd mode),
mkfs -m pool-root pinning gated on CONF_ZFS_OPENZFS, cpiod umount prefix
normalization so the host-side image build flushes the pool, and the ZFS
validation/benchmark test suite.
The conf_zfs=openzfs build linked the legacy BSD-ZFS kstat stub (bsd/.../opensolaris_kstat.o) to satisfy kstat_create/install/delete, but every OpenZFS module was compiled against the OSv SPL kstat_t in external/openzfs/include/os/osv/spl/sys/kstat.h (~64 bytes: ks_data, ks_ndata, ks_data_size, ks_flags, ks_update, ks_private, ks_private1, ks_lock). The BSD stub allocated only a 16-byte kstat_t, so OpenZFS callers such as arc_init()/dnode_init() wrote ks_update/ks_private past the end of the allocation, corrupting the malloc free list. The next kstat_create() then returned a garbage pointer and faulted at ks_ndata = ndata (SIGSEGV -> abort with empty backtrace right after the version banner, before mkfs ran). Fix: implement OSv-native kstat_create/install/delete in openzfs_osv_compat.c using the correct OpenZFS kstat_t layout (virtual kstats; OSv has no /proc or sysctl consumer), and filter the BSD opensolaris_kstat.o out of the openzfs solaris object list in the Makefile so only the correct implementation is linked. The zfs_builder guest now boots, runs mkfs (creates + mounts pool osv), populates it via cpiod, and exports cleanly.
…mount zfs_unmount() (called by zfs destroy, zpool export, and property remounts) only unmounts a dataset if it can find it via libzfs_mnttab_find() -> libzfs_mnttab_update() -> getmntent(). On OSv that lookup went nowhere for two reasons: - MNTTAB pointed at a nonexistent path, so fopen(MNTTAB) failed before getmntany() ever ran; and - the getmntent/getmntany stubs in libzfs_util_os.c returned EOF. Datasets auto-mounted by the kernel (zfs_domount at pool/dataset create time, not through the libzfs do_mount path) were therefore invisible to libzfs, the objset stayed owned, and zfs destroy / zpool export failed with EBUSY / 'dataset is busy'. Fix, as two edits to the OpenZFS patch series: - patch 0004: MNTTAB -> /etc/mnttab (created empty at boot, so fopen succeeds and libzfs proceeds to getmntany()). - patch 0020: add lib/libzfs/os/osv/libzfs_mnttab_os.cc implementing getmntent/getmntany over the live VFS mount table via osv::current_mounts() (filtered to ZFS), replacing the EOF stubs. Wire the new C++ shim into the libzfs build, filtering the C-only warning flags out of ozfs-cflags-common that a C++ translation unit rejects. Makes zfs destroy and zpool export/import work on OSv, which the feature-coverage tests depend on.
zpool export/import intermittently aborted with Assertion failed: owner.load(...) == sched::thread::current() (core/lfmutex.cc: unlock: 221) in condvar::wait(). Root cause: the OpenZFS userland thread pool (lib/libtpool) hands jobs to worker pthreads that block in pthread_cond_wait() on a shared tp_mutex. On OSv a pthread mutex and condvar ARE the kernel lockfree::mutex + condvar, whose wait-morphing protocol transfers mutex ownership from the signalling thread to a waiter. During tpool_destroy() teardown that handoff races: a worker returns from pthread_cond_wait() and re-enters it, unlocking a tp_mutex it no longer owns -> lfmutex owner assertion. Reproduced deterministically on zpool import, whose device-scan (zutil_import.c) and mount (libzfs_mount.c) thread pools default to hundreds of workers (mount_tp_nthr = 512). Fix: run tpool jobs synchronously on OSv (0021) and force serial dataset mounting (0022). OSv threads are cheap and these jobs are short, so serial execution removes the worker/teardown machinery and the race at no practical cost. Verified 0/8 asserts (was 8/8) across create/export/ import/status cycles from a clean patch-series rebuild.
…dataset) Patch 0017 reclaimed only the mount-root znode before dmu_objset_disown. A mounted child dataset (e.g. pool/fs) leaves its own znodes live, each holding an object bonus buffer that pins a dnode. dnode_destroy never runs, the objset os_dnodes list never empties, and spa_export() blocks forever in spa_evicting_os_wait(). Reproduced deterministically by: zpool create; zfs create pool/fs; zpool export (hang). Walk z_all_znodes at unmount and inactivate every remaining znode, as vop_inactive would, so the objset drains. Verified 6/6 completions (was 0/6, hung) for create+child-fs+export+import+destroy.
OSv read_partition_table() names MBR slots 0-based (first slot is /dev/vblkN.0) and exposes a raw unpartitioned disk directly as /dev/vblkN with no child node. zfs_append_partition() unconditionally appended .1, so \x27zpool create test /dev/vblk1\x27 on a raw wiped NVMe failed with \x27cannot open /dev/vblk1.1: No such file or directory\x27. Fix (patch 0023): only append .1 when that partition node actually exists; otherwise use the whole raw disk as-is, matching Linux/FreeBSD whole-disk behavior (ZFS writes its own labels to the whole device). Also adds FINDINGS-osv-openzfs.md documenting Bug 1, Bug 2, and the page-allocator-under-memory-pressure investigation (>=4G runtime guests). Verified: zpool create on a raw disk now succeeds, pool ONLINE.
…r 0) zpool list faulted with strlen(NULL) in print_line() because zpool_prop_column_name() returned NULL for a default list column on OSv. The pool data row is correct; only the header label resolves NULL. Guard header against NULL, falling back to the ZFS missing-value glyph, so the command cannot crash.
zfs_is_readonly() was hardcoded B_FALSE, so a readonly=on dataset stayed writable on OSv. Read the readonly property at mount into a new zfsvfs->z_readonly, return it from zfs_is_readonly(), and reject the write vnops (write/create/remove/rename/mkdir/rmdir/setattr/truncate/symlink) with EROFS. Verified writes fail EROFS after remount and readonly=off restores writability.
vdev_disk_open() never set vd->vdev_has_trim, so zpool trim/autotrim reported trim not supported. Enable it optimistically (OSv has no capability query): the ZIO_TYPE_TRIM->BIO_DISCARD path reclaims space on virtio-blk with discard, and devices without discard get ENOTSUP mapped gracefully. Verified zpool trim completes 100% with discard=unmap.
…ryption/dedup/draid/TRIM/O_DIRECT/etc)
Head-to-head storage microbenchmark of the two ZFS ports on identical a bare-metal host, 8 GiB KVM guest, single-vdev (raw NVMe) and raidz2 (7x20 GiB). No Postgres/HammerDB - isolates the filesystem layer. Key finding (the ARC-bridge measure): mmap of a 512 MiB file costs BSD-ZFS ~5 MiB extra RAM vs OpenZFS ~501 MiB - BSD's unified ARC/page-cache bridge shares pages, OpenZFS double-buffers via its borrow path. Same root cause drives BSD's warm-read and cached-random-read wins. OpenZFS wins the write/CPU paths (seq write +34%, lz4 +56%, metadata +63%, fsync +11%) and uniquely offers O_DIRECT (ARC bypass, ~raw-ceiling, BSD has no direct=). Harness (scripts/bench/): pure-C zfs-bench.c (a g++ .so pulls libstdc++ symbols OSv can't resolve at load; C avoids GLIBCXX/CXXABI/__isoc23). Runs off the zfs_builder bootfs - no usr.img cpiod populate. rebuild-bench.sh, run-bench.sh, run-all.sh drive it; results.tsv is the raw data. Copyright (C) 2026 Greg Burd
…en_zfs/ Per review: external/ is reserved for things compiled into the kernel (libfdt, acpica). The vendored OpenZFS tree is a module dependency, so move the submodule from external/openzfs to modules/open_zfs/openzfs alongside its patch series and module.py. Updates: .gitmodules path (still points at the published upstream github.com/openzfs/zfs at tag zfs-2.4.2, no fork), the OZFS/OPENZFS make variables (Makefile + bsd/sys/cddl/openzfs_sources.mk), the parse-time patch-apply step (stamp path, -d guard, and the git -C ... apply path, whose relative patch prefix changes from ../../modules/open_zfs/patches to ../patches now that the submodule sits one level deeper), and the doc comments that named the old path.
The per-child COW address space (clone_pt_level<1>) shared any 2 MB large
page verbatim between parent and child instead of copy-on-writing it:
if (ppte.large()) { child_pt[i] = ppte; continue; } // 2MB: share as-is
so a forked child writing a private large mapping scribbled the parent(s)
too. This hit large malloc (>= 2 MB, mmap-backed), large MAP_ANON regions,
and any 2 MB-backed private writable mapping in the application slots.
Fix: for a WRITABLE large PD entry, split it to 4 K in the parent in place
(the existing split_large_page primitive) and fall through to the normal 4 K
clone path, which already makes the correct per-page decision -- private
writable -> COW, MAP_SHARED -> stays shared, read-only -> shared as-is. A
read-only large page can never diverge, so it is still shared verbatim. This
reuses the whole 4 K COW fault machinery (clone_pt_level0 +
handle_cow_write_fault); the only cost is one 4 K page table per split large
page, acceptable under the opt-in CONF_fork.
Verified on x86-64 (KVM microvm):
- a standalone musl-PIE fork test now isolates a private >2 MB malloc across
fork (child write does not change the parent; parent write does not change
the child), while a MAP_SHARED region stays mutually visible;
- tst-fork 10/10, tst-fork-cow 6/6, tst-fork-deep and tst-execve all pass
(no regressions);
- conf_fork=0 builds clean with the whole fork path (this change included)
compiled out (clone_address_space absent from loader.elf).
Note: this fixes large-page (application-slot) COW. OSv's small-object
malloc heap lives in the shared kernel identity map (mem_area::mempool), which
fork() shares by design; isolating that is a separate change.
All gated behind CONF_fork (default off).
Copyright (C) 2026 Greg Burd
Stock fork-per-backend PostgreSQL corrupts memory on OSv because OSv's small-object malloc heap lives in the kernel identity map (mem_area::mempool, VA >= 0x400000000000), which is shared verbatim across every address space and cannot be copy-on-write cloned on fork(). A forked child writing to a heap object inherited from its parent scribbles the SHARED heap. Give application allocations a home in an ordinary anonymous mmap region in the app-slot VA range (below phys_mem), so clone_address_space()'s existing COW machinery isolates the whole app heap per child. virt_to_phys() already page-walks addresses below phys_mem, so arena pages stay DMA-usable. The arena is a segregated free-list whose bookkeeping lives entirely in kernel BSS (never in arena pages), so managing it never faults an arena page and never recurses into malloc during fork's own page-table work. std_malloc routes an app thread's allocations to the arena (decided by is_app at malloc time); free()/realloc() dispatch purely by address range, so cross-thread alloc/free is always handled by the allocator that owns the address. All gated behind CONF_fork (default n): with fork disabled none of this is compiled and the heap behaves exactly as before. tst-pgfork.c proves a forked child's writes to inherited AND new small/large heap objects do not leak into the parent. Author: Greg Burd
Progress on the fork heap arena: - arena allocator reworked to use a preemption-disabling spinlock (not a sleeping mutex, which is illegal when malloc runs preempt-disabled) with the page-touching header write done outside the lock. - clone_address_space() pre-reserves the child address_space object and moves the child vma-list construction (anon_vma allocations) OUT of the vmas_mutex critical section, and snapshots parent vmas under the lock into a pre-grown vector -- avoiding malloc under the fork lock (the malloc-during-fork trap). - sched::thread objects and coherent wait_records forced to the identity kernel heap (kernel_heap_scope) so cross-address-space kernel structures are not COW-private. Known blocker (documented in /tmp/fork-arena-result.txt): with the arena active, the first fork() still hits an illegal page fault (page_fault: preemptable() assert) during/around clone -- a COW write fault occurring in a preemption/irq-disabled window. Arena allocator validated standalone (tst-realloc passes). conf_fork=0 verified clean (no fork_arena symbols, boots). Author: Greg Burd
… the fork lock Further hardening of the fork heap arena (still gated on CONF_fork, default n): - arena allocator is now fully lock-free (Treiber-stack free lists + atomic bump pointer): no preemption-disabling lock is held while touching a chunk, so a post-fork copy-on-write page fault on a chunk is always serviceable (OSv forbids faulting with preemption/interrupts disabled). - clone_address_space(): defer flush_tlb_all() and the live_child_address_spaces bump until AFTER releasing vmas_mutex, and build the child vma_list from a snapshot outside the lock -- a write that COW-faults a kernel .bss global (or any malloc) must not run while vmas_mutex is held for write, or the COW fault handler self-deadlocks re-acquiring it. Remaining blocker: COW-cloning the arena subtree specifically still triggers an illegal (preemption-disabled / recursive) page fault at the first fork(); sharing the arena instead lets fork() run (5/5 early tst-fork checks) but loses heap isolation. Documented in /tmp/fork-arena-result.txt. Author: Greg Burd
A thread's TLS block and tiny/large syscall stacks hold state the kernel reads and WRITES from preemption/interrupt-disabled contexts (TLS __thread vars; the kernel runs syscalls on the syscall stack). If they were allocated from the COW fork arena (an app thread creating a thread after the arena is live), fork would write-protect them and the next preempt-disabled write would take an illegal COW page fault. Force these allocations to the identity kernel heap via kernel_heap_scope, matching the thread-object and wait_record treatment. Author: Greg Burd
…cycle + execve-continuation allocs to identity heap Root cause (found via qemu+gdb hardware watchpoint): the kernel_heap_scope guard around aligned_alloc in setup_tcb was being ELIDED by the compiler. force_kernel_heap was a plain __thread unsigned; GCC models aligned_alloc (and operator new) as not reading global memory, so the paired ++/-- around a single such call was dead-code-eliminated -> app-thread TLS/thread objects landed in the COW arena -> writing preempt_counter (in a COW arena page) during fork triple-faulted. Fixes (all gated CONF_fork): - force_kernel_heap -> volatile, so the scope's inc/dec survive around builtin-modeled allocators (verified in disassembly + a minimal repro). - wrap the whole sched::thread construction in make() in a kernel_heap_scope so the thread object, _detached_state, and _wakeup_link._helper (all touched by the scheduler cross-AS with preemption off) go to the identity heap. - __cxa_thread_atexit_impl linked_destructor nodes -> identity heap. - execve() continuation strings (s_path/s_args/s_env): allocate their backing storage under kernel_heap_scope so it survives destroy_address_space() of the child COW AS (was reading a freed arena buffer -> 'executable too short /'). Validated: tst-pgfork PASS 9/9 (child small-heap writes do NOT leak to parent - the arena's core purpose works). tst-fork now passes 4/10; fork+execve (test 2) still hits a nested page-fault under investigation.
…hes the arena A forked child's fork+execve (tst-fork test 2) aborted with 'exception_depth <= 1' during ELF demand-paging. The exec'd program's file-backed pages fault in through vm_fault -> pagecache/ROFS, which allocates cached_page bookkeeping. On an app thread those allocations route to the COW fork arena; first-touching a fresh arena page WHILE servicing a fault (already holding vma_list_mutex, possibly one exception deep) takes a second page fault, exceeding exception_depth and asserting. vm_fault services KERNEL work: pagecache pages and filesystem demand-paging buffers are shared kernel infrastructure that must be identity-mapped and never a COW arena page. Wrap the whole handler in a kernel_heap_scope (gated CONF_fork) so no arena page is allocated or first-touched during fault servicing. Restores tst-fork 10/10 with the arena enabled.
…loc)
The detached-thread reaper kept its pending zombies in a std::list<thread*>,
whose push_back() in add_zombie() allocates a list node. Two problems with
that under the fork arena:
1. add_zombie() runs from a terminating thread. For a fork child that is an
APPLICATION thread executing in the child's COW address space, so the
std::list node is malloc'd into the fork arena -- landing on a
COW-private page at an arena VA. The reaper drains the list from AS0,
where that same arena VA resolves to a DIFFERENT physical page: it reads
a garbage 'thread*' and join()s a bogus pointer forever. OSv then never
runs the child's cleanup, the child's application_runtime reference never
drops, application::join()'s wait_until(_terminated) never completes, and
the guest hangs at shutdown instead of powering off.
2. The node allocation can also fault a fresh heap page from a
preemption-disabled termination window.
Switch _zombies to a boost::intrusive::list threaded through a new
thread::_zombie_link hook. The link lives inside the thread object (identity
kernel heap), so add_zombie() performs NO allocation: coherent across address
spaces and safe from a non-preemptable context.
… heap async::run_later()/timer task nodes (one_shot_task, percpu_timer_task) were allocated from the COW fork arena. The async worker walks its intrusive task list with preempt_lock held (preemption disabled); touching a COW arena page there faults, and OSv forbids page faults in non-preemptable context (assert(sched::preemptable()) in page_fault). Force these nodes onto the identity kernel heap via fork_arena::kernel_heap_scope, matching the existing wait_record / child-registry kernel-heap pattern. Surfaced by booting stock musl PostgreSQL 18 with CONF_fork: the postmaster's first forked child (checkpointer) tripped this the moment the async worker fired a TCP-stack run_later callback. Clears that wall; shared-memory attach across fork() then works (verified: child reads PG ProcGlobal shared state).
fork_arena::init() reserved the 512 MiB arena VA with map_anon(mmap_fixed) WITHOUT populating it, so pages faulted in lazily on first touch. Under real concurrent PostgreSQL load fork_arena::alloc() gets entered from an IRQs-off / preemption-off context (mid-exception, kernel work), and first-touching a freshly bump-carved page there DEMAND-FAULTS -- which page_fault forbids (assert(ef->rflags & rflags_if) / assert(preemptable)) -> abort. Add mmu::mmap_populate to the map_anon flags: all 512 MiB is RAM-backed at init, so alloc's first write always hits a present page and can never fault. That makes alloc safe to call from ANY context. clone_address_space() still COW-clones the whole vma per child; the child only WRITE-faults (COW break) from app context with irqs/preemption on, so the original fault-context invariant is preserved on the child path. 512 MiB fully backed is acceptable for a per-app fork heap; boot is unaffected (~3-4 s) and the whole fork suite stays green (tst-fork 10/10 x5, tst-pgfork 9/9, tst-fork-cow/deep/execve pass). All gated CONF_fork; conf_fork=0 unchanged (arena compiles to an empty TU, exits 0). Also bump apps to the pg18-fork test module (musl PG18 fork-per-backend image).
…tion bug A fork child that spins deep on its same-VA stack while being heavily preempted (touching .data/.bss globals, its own stack, FPU/SSE, and taking syscalls + COW faults) has its register context corrupted on the preemptive context-switch resume path -- a callee-saved register (rbp) is reconstructed from a .text immediate and a pointer resolves into the same-VA stack, giving a page fault outside the application. Deterministic on -smp 1 and -smp 2. The existing fork tests only fork short-lived children preempted a handful of times, so they never exercised this window. The manifest entry is auto-derived from the tests list; the test itself is unconditional (fork() just fails when built conf_fork=0). Author: Greg Burd
…eempt test + mmap-in-child repro Three related changes from investigating the PostgreSQL fork-child wall: 1. libc/signal.cc: the global "waiters" std::list (one per signal, populated by wait_for_signal / drained by unwait_for_signal) is manipulated cross-address- space -- a fork child and its parent both push/remove on the same shared list. Its list nodes were allocated from the COW fork arena, so a child's node became a COW-private page whose next/prev links were inconsistent with the shared sentinel seen from the other AS; a later list::remove() traversal (reached via sigprocmask in the checkpointer child) followed a dangling link and page-faulted with interrupts disabled -> assert(rflags & IF) in page_fault. Force the identity kernel heap for those node alloc/free, the same rule already applied to thread objects and wait_records. Proven with gdb: this was one of the two PG crash variants after "InitAux shmem read OK"; the fix eliminates it. Gated CONF_fork. 2. tests/tst-fork-preempt.cc: reframed honestly. A long-lived fork child spun deep on its same-VA stack while heavily preempted (compute + .data/.bss globals + stack + FPU + syscalls) PASSES -- proving the preemptive context switch itself preserves a fork child's register/stack context (this refutes the earlier hypothesis that the deferred-CR3 same-VA switch was the wall; see report). Green on -smp 1 and -smp 2. 3. tests/tst-fork-child-mmap.cc: standalone repro (NOT in the suite -- it aborts the unikernel) of a separately-found real bug: a fork child's own mmap() then write SIGSEGVs, because only the page-FAULT path is per-child-AS aware while the mmap/munmap ALLOCATION path (mmu::allocate / map_anon via the global vma_list + vma_range_set) is not. See /tmp/pg-preempt-fix.txt for the full analysis. Author: Greg Burd
The OpenZFS image-populate step (create_zfs_filesystem in scripts/build and the
cpiod builder in scripts/upload_manifest.py) OOMs at 512M when building the
PG-on-ZFS image; 4G is fine on the build host. scripts/build now honours an
optional ${zfs_builder_mem} override (default 4G).
Signed-off-by: Greg Burd <greg@burd.me>
…so an AS0 writer can wake a forked-backend reader rwlock::rlock() registers a pending reader as a lockfree::linked_item<thread*> that is a LOCAL on the waiting thread stack, pushed onto the shared _read_waiters queue. With per-child COW stacks, when a forked PostgreSQL backend blocks as a reader its read_waiter lives at a COW-private app-stack VA (0x200000...); an AS0 writer releasing the lock walks _read_waiters in wake_pending_readers() and dereferences that VA through AS0 page tables -> SIGSEGV (Aborted in rwlock::wake_pending_readers+82 <- rwlock::wunlock <- taskqueue_thread_loop), which crashed a pgbench bulk COPY at ~20%. Mirror the existing lfmutex/condvar coherent_wait_record mechanism: when fork_child_needs_heap_wait_record() (a forked-child caller, or any live child AS while the AS0 parent waits), allocate the linked_item from the identity kernel heap (fork_arena::kernel_heap_scope) so its VA is coherent in every address space; the waiter frees it after waking. Default OSv / AS0-only keeps the zero-overhead on-stack fast path. Gated CONF_fork; conf_fork=0 byte-identical. Verified: pgbench -i -s 50 (5,000,000-row bulk COPY) now completes 100% where it previously aborted at 20%. Signed-off-by: Greg Burd <greg@burd.me>
…y kernel heap so a forked backend's rename survives cross-AS drele() dentry_alloc() already wraps its dentry + d_path allocations in a fork_arena::kernel_heap_scope so shared dentry-cache infrastructure stays off the per-address-space COW fork arena. dentry_move() (the rename path) did NOT: its dp->d_path = strdup(path) ran unguarded, so a rename issued by a forked PostgreSQL backend allocated the new d_path in that backend's COW-private fork arena. When the last reference to that dentry later dropped in a DIFFERENT address space (AS0 postmaster or a sibling backend), drele() -> free(dp->d_path) routed the pointer to fork_arena::free(), which read the chunk header at that arena VA through the freeing AS's page tables. After the COW divergence that VA holds a different physical page there, so the header magic mismatched and recover() aborted: Assertion failed: h->magic == chunk_magic (core/fork_arena.cc: recover: 257) cloudius-systems#3 fork_arena::free cloudius-systems#4 drele (dp=...) at fs/vfs/vfs_dentry.cc:229 <- free(dp->d_path) cloudius-systems#8 vfs_file::close <- close(fd) PostgreSQL triggers this during a checkpoint that recycles WAL segments (rename of a pg_wal file), which is exactly the "N recycled" checkpoint that crashed a forked backend on the RAID-Z pool. Fix: wrap dentry_move()s d_path strdup in the same kernel_heap_scope as
… AS0 so the postmaster launches parallel workers (parallel-query hang) A forked PostgreSQL backend that signals the postmaster used kill(pid, sig) with pid == OSV_PID (2) -- the postmaster is the top-level app / AS0, whose getpid() is OSV_PID. The CONF_fork kill() path resolved a cross-process target via osv::fork::as_for_pid(pid) and returned ESRCH when that was null. AS0 is never entered into g_pid_as (only fork children are register_pid()'d), so as_for_pid(OSV_PID) == nullptr and kill(OSV_PID, sig) from a child returned ESRCH -- the signal was dropped. That is the lost wakeup behind the parallel-query hang. A parallel-query leader (itself a forked backend) launches workers via RegisterDynamicBackgroundWorker, which sets a shared request slot and then does kill(PostmasterPid, SIGUSR1) to make the postmaster fork the workers. With the signal dropped (ESRCH), the postmaster -- parked in its ServerLoop WaitEventSetWait/epoll_wait -- never sees the request, never forks a worker (debug1 logging showed ZERO "starting background worker" lines), and the leader waits forever in WaitForParallelWorkersToAttach. Block I/O was fully drained; this is a pure coordination lost-wakeup, same cross-AS/per-fork-child-pid routing class as the shipped SIGCHLD/SIGURG-latch fixes. (It also explains the noted SIGHUP-to-postmaster ESRCH from pg_reload_conf().) Fix (libc/signal.cc, #if CONF_fork): exclude OSV_PID from the cross-process target resolution -- treat it like the self/broadcast case so target_as stays nullptr and the handler runs in AS0 (mmu::kernel_address_space()), which is exactly where the postmaster's handler must run to poke its own latch/self-pipe and wake its ServerLoop. Never returns ESRCH for the top-level app. The later (pid == OSV_PID) wake_up_signal_waiters()/handler-in-AS0 logic is unchanged. conf_fork=0 does not compile this block (byte-identical). Validated: with max_parallel_workers_per_gather=2, a parallel Seq Scan Gather `select count(*) from pgbench_accounts` returns 10000000 (== the non-parallel reference), and EXPLAIN ANALYZE shows "Workers Planned: 2 / Workers Launched: 2" with both workers processing rows. Previously: 0 workers launched, hang. Signed-off-by: Greg Burd <greg@burd.me>
…_simple + zfs_space on OSv so ZIL replay works (sync=standard durability)
The OSv OpenZFS platform layer left zfs_write_simple() and zfs_space() as
ENOTSUP stubs. Both are on the ZIL replay path that runs on pool import after
a crash when a dataset has sync=standard (ZIL active, here on a SLOG vdev):
zfs_replay_write() (TX_WRITE, txtype 9) -> zfs_write_simple()
zfs_replay_truncate() (TX_TRUNCATE, txtype 10) -> zfs_space(F_FREESP)
Consequence on the RAID-Z (4xEBS raidz1 + NVMe SLOG + L2ARC): a kill -9 mid-txg
with sync=standard, then re-import, logged
"ZFS replay transaction error 95 [ENOTSUP], dataset pgdata, txtype 9/10"
and dropped every logged intent-log record (ZIL gave no durability); an earlier
build variant instead SPL-PANICked in abd_alloc_linear
(VERIFY3U(size <= SPA_MAXBLOCKSIZE)) on a log record it could not replay.
Either way sync=standard was not usably crash-recoverable through the ZIL on OSv
-- forcing sync=disabled and losing the SLOG's purpose.
Patch 0029 implements both the OSv way, reusing primitives that already work
here (so this is a single-process correctness fix, no fork/COW dependency,
correct for every config -- deliberately NOT CONF_fork-gated):
* zfs_write_simple: one-iovec struct uio over the kernel buffer +
zfs_write(zp, &zuio, O_SYNC, NULL) (the zfs_vop_write shape / FreeBSD's
vn_rdwr(UIO_SYSSPACE, IO_SYNC)); refresh vp->v_size from zp->z_size.
* zfs_space: F_FREESP wrapper delegating to the already-present
zfs_freesp() (zfs_znode_os.c), identical to FreeBSD's zfs_space.
As an upstream-openzfs source change this is a modules/open_zfs/patches/ entry
(git-apply'd to the pinned submodule at build time), NOT a submodule edit.
Validated on the RAID-Z: sync=standard, insert marker + checkpoint + settle,
kill -9, reboot -> NO replay error, NO panic, ZIL TX_WRITE/TX_TRUNCATE replay,
PG crash recovery completes, and the row survives crash+reboot.
Signed-off-by: Greg Burd <greg@burd.me>
…bling fork backends (PG shared_buffers catalog-read-zero) Stock PostgreSQL (shared_memory_type=mmap, the default) puts shared_buffers in one big anonymous MAP_SHARED segment the postmaster creates BEFORE forking backends. Only a few of its pages are touched pre-fork; the system-catalog index/relcache pages are first-faulted by an individual backend AFTER its fork. On OSv this produced the multi-backend catalog-read-zero wall: backend cloudius-systems#1 reads a catalog-index page fine, but sibling backends cloudius-systems#2+ (zero writes in between) read the SAME shared page as ZEROS ("pg_authid_rolname_index contains unexpected zero page", "cache lookup failed"), so HammerDB can't even authenticate vuser cloudius-systems#2. Hypervisor-independent (KVM and Firecracker identical). Root cause (core/mmu.cc): an anon MAP_SHARED page not present in the parent's page table at fork time is cloned as an EMPTY child PTE (clone_pt_level0 can only share what is already present). When a sibling backend later faults that VA, initialized_anonymous_page_provider::map() calls memory::alloc_page() and installs a FRESH, PRIVATE, ZERO page in that backend's AS only. Every AS that first-touches the page post-fork gets a different private zero page -- they never converge on the ONE physical page MAP_SHARED promises. Fix: back an anon MAP_SHARED vma with a shared_anon_page_provider that keys a process-global registry (identity kernel heap, like shm_file::_pages) on the absolute page VA. An anon MAP_SHARED region lives at the SAME VA in every fork AS (clone_address_space keeps identical VAs), so the VA uniquely identifies the shared page: the first AS to fault it allocates+records it; every later AS (sibling backend or parent) that faults the same VA maps THAT frame. All siblings converge. The provider tags the PTE pte_shared so per-AS teardown (free_child_pt_level0) does not free the jointly-owned frame -- without this, the first backend to exit freed the shared page and later siblings read freed/zeroed memory (the "second backend reads zeros" symptom). A searched (addr==NULL) mapping is relocated by allocate() after construction, so the provider base is re-keyed via anon_vma::update_shared_base() once the real VA is known. Regression test tests/tst-fork-shared-catalog.cc mirrors PG exactly: parent creates an anon MAP_SHARED region, forks THREE siblings; child A writes a page, siblings B and C (and the parent) must read A's data, not zeros. Reproduces the zero-read on HEAD; PASSES with the fix at -smp 2 and -smp 4 (5/5 runs, real concurrency, not gdb-serialized). Full fork suite still 0 failures (tst-fork/cow/deep/child-mmap/file-mmap/posix-shm/shared-write/shared-race/ serial/sigchld-latch/arena-freelist/irq-cow/preempt/socket/conn-socket/ timer-park-stress). All #if CONF_fork; non-fork byte-identical. Author: Greg Burd <greg@burd.me>
…kq quiescence OSv taskq_wait() enqueued a single barrier task and drained only that barrier. With an 8-worker system_taskq a free worker runs the barrier while the other 7 are still mid dnode_sync/dbuf_sync, so taskq_wait returned early and the syncing thread raced the still-running workers -> dbuf/dirty-record/refcount corruption (arc_write_done VERIFY3S refcount underflow + lfmutex owner-mismatch panics). ZFS documents this contract (dmu_objset.c PORTING note). Add taskqueue_drain_all() which waits until the queue is empty AND no worker is active (illumos semantics), and wire taskq_wait() to it. conf_fork unaffected.
…tity linear map + heap-allocate reclaimer waiter node for fork children Under CONF_fork, ZFS large allocations (>= huge_page, or the non-contiguous fallback) made under fork_arena::kernel_heap_scope (force_kernel_heap) went to map_anon(), which lands in the CURRENT thread address space app mmap slot (VA 0x2000..). A forked PostgreSQL backend making such an allocation (e.g. an 8k zio_buf / dbuf db_data) put the VA only in that child page tables; AS0 txg_sync / zio-completion threads and sibling backends faulted outside application on db_data. Force force_kernel_heap large allocations onto free_page_ranges (linear map, phys_mem 0x4000.., a kernel PML4 slot mapped verbatim in every AS) and never fall back to map_anon. Verified buffers move 0x2000 -> 0x4000. Companion: reclaimer_waiters::wait() wait_node is a stack local dereferenced by the AS0 reclaimer; for a fork-child waiter its COW-private stack VA GPFd wake_waiters(). Heap-allocate the node for fork children (identity, coherent), same pattern as coherent_wait_record. Non-fork path byte-identical. Signed-off-by: Greg Burd <greg@burd.me>
Describe the test host generically and reword internal comment tags to plain notes. No functional change.
…-swap asm (no post-swap %rbp stack read)
The fork COW context switch (switch_as branch) re-tested switch_as and reloaded
the FPU control words (fpucw/mxcsr) in C++ AFTER the inline asm swapped rsp/rbp
AND CR3 to the incoming fork child. The compiler spills switch_as and the
fpucw/mxcsr locals to %rbp-relative stack slots, and %rbp at that point already
points at the incoming child stack under the just-loaded child CR3. So the
post-swap code
mov -0x50(%rbp),%rdx ; cmp %rdx,-0x48(%rbp) (the switch_as re-test)
movzwl -0x52(%rbp),%eax ; mov %ax,-0x34(%rbp) (the else-branch reload)
reads and writes the child COW stack in an irq-off, non-preemptable window. A
COW write-fault there trips assert(sched::preemptable()) at arch/x64/mmu.cc:38,
and a mis-read switch_as takes the wrong branch and can ldmxcsr a garbage MXCSR
(reserved bits set) -> #GP, so the resumed child never runs again (a forked
backend parks in switch_to and is never rescheduled). The prior FPU fix
(1ae2602f9) hardened the reloaded VALUES for the switch_as==true path but left
the branch decision and the else-branch operands as %rbp-relative reads, still
resolved through the incoming child AS. Dormant when the surrounding code
compiles exactly like c8f9c82b; exposed once code layout shifts the fork timing.
Fix: emit emms + fldcw + ldmxcsr from kernel-identity-mapped .rodata statics
INSIDE the switch_as asm, right at the 1: resume label, then return. RIP-relative
.rodata is identically mapped in every address space (never COW), so nothing after
the CR3 swap touches a %rbp-relative (stack) slot: no COW fault, no mis-read. The
canonical control words are correct for every OSv thread (issue cloudius-systems#1020; switch_to
does no fxsave/fxrstor, so no per-thread FPU state is preserved across a switch
regardless), so this is semantically identical to the old fldcw(fpucw)/ldmxcsr(mxcsr)
for the AS-crossing case. The shared post-asm reload now runs only for the
non-crossing (same-AS) path, whose %rbp is the resumed threads own coherent stack.
…e self-pipe coherent Two coupled CONF_fork fixes so a forked PostgreSQL backend's own latch and ProcSignalBarrier self-signals reach it, without breaking the child-exit notification. kill(): a fork backend's OWN SIGURG (latch) and SIGUSR1 (ProcSignalBarrier) self-signal must run its handler in the caller's own copy-on-write address space, so the handler sets the pending flag / pokes the self-pipe in the copy the backend actually reads. Restrict the self-routing to SIGURG/SIGUSR1: the child-exit SIGCHLD that fork emulation raises with kill(getpid(), SIGCHLD) from a dying child must NOT be diverted, or the parent's reaper runs against the child's torn-down mapping and faults, and the parent never reaps the exited child (it then never finishes startup). as_for_pid() also gates to a live registered child, so a child mid-teardown falls back to the top-level table. pipe2(): allocate the pipe_buffer and pipe_file on the identity kernel heap so the self-pipe backing a latch is coherent across every fork address space (a cross-backend wakeup writes the pipe from another address space while the waiting backend polls the same pipe). Same rule as the epoll and file containers. Both are inside #if CONF_fork, so a non-fork build is byte-identical. Validated: with the routing present, boot-to-ready 10/10 on a fresh recordsize=8k pool; CREATE DATABASE + DROP DATABASE completes 5/5 (was an indefinite wait for the emitter's own ProcSignalBarrier); pgbench -c8 0 failed transactions.
Two independent bugs prevented building the fork-enabled kernel
(conf_fork=1) on GCC 11.x, and also broke a clean conf_fork=0 build on
both GCC 11 and GCC 13:
1. core/mmu.cc defined anon_vma::~anon_vma() unconditionally, gating only
the body with #if CONF_fork, while include/osv/mmu.hh declares the
destructor only under #if CONF_fork. When CONF_fork is 0 the class
does not declare the destructor but the .cc still defines one, which
is an ISO C++ violation ("definition of implicitly-declared
destructor") rejected by both GCC 11 and GCC 13. Move the #if CONF_fork
to wrap the whole destructor definition so it matches the declaration.
2. Makefile auto-generates the kernel configuration with
$(shell make -f conf/Makefile -j1 config), which did not forward
conf_fork. GNU make does not propagate a command-line variable
override into a $(shell ...)-invoked sub-make, so the Kconfig option
(def_bool driven by the conf_fork env var) always resolved to
CONF_fork=0. A conf_fork=1 build therefore produced CONF_fork=0 and
silently failed to compile the fork sources. Forward conf_fork to the
config sub-make.
Both changes are gated on CONF_fork / conf_fork. conf_fork=0 output is
unchanged: at CONF_fork=1 the mmu.cc change is a preprocessor no-op
(the -g0 core/mmu.o is byte-identical), and forwarding an empty conf_fork
yields the same generated config as before.
Verified: fork+ZFS kernel compiles and links clean and the resulting
image boots (ZFS root mounted) on both GCC 11.5 and GCC 13.3.
Signed-off-by: Greg Burd <greg@burd.me>
…o a forked backend does not fault in elf::resolve_pltgot OSv resolves a dynamic object's PLT/GOT lazily: the first call to an imported function traps into elf::object::resolve_pltgot(), which looks the symbol up and writes the resolved address into the object's .got.plt slot. Under CONF_fork a forked child runs in a copy-on-write clone of the parent address space, and two facets of the lazy resolver are not coherent across that clone, which surfaces as a NULL dereference inside elf::resolve_pltgot in a forked child (e.g. a PostgreSQL backend under a heavy stored-procedure workload). Root cause and fix, both CONF_fork-gated (conf_fork=0 emits identical code): 1) COW-stale GOT. The writable file-backed segment that holds .data/.got/.got.plt is demand-paged. The dynamic linker RELOCATES that GOT in memory at load (object::relocate / relocate_pltgot rewrite each slot by the object base and install the resolver trampoline pointer and the object pointer in pltgot[1]). Those writes diverge the page from its on-disk bytes. If such a relocated page is not resident when clone_address_space() clones the parent, the child inherits an empty leaf PTE and its first access re-reads the ORIGINAL, unrelocated bytes from the file (pltgot[1]=0, raw jump slots). The child's lazy resolver then dereferences a NULL elf::object and faults. Fix: mmap_populate writable file-backed segments at load so every relocated page is resident in the parent; clone_address_space then COW-shares the relocated pages and the child always reads the correct GOT. OSv does not swap and a written file-private page becomes anonymous, so a populated page stays resident through fork. 2) Symbol-set node in the child COW arena. When a forked child is first to resolve a cross-object slot, resolve_pltgot() inserts into the shared elf::object's _used_by_resolve_plt_got set. Without care the set node is allocated from the child's private COW arena while the set lives on the shared identity heap; a sibling or the top-level address space later walking that set dereferences a node mapped only in the child's (possibly already reclaimed) address space. Fix: allocate the set node under a kernel_heap_scope so it lands on the identity heap, coherent across every fork address space. Adds tests/tst-fork-pltgot.cc: a forked child is first to resolve a batch of libc/libm PLT symbols the parent never called, plus a parent-and-reaped-children cross-object stress of the shared symbol set. Passes at -smp 4 and -smp 8; boot-to-ready 10/10 unaffected.
…orked child runs the lazy resolver (fixes the fork symbol-lookup wall) Under CONF_fork, force bind_now=true in object::relocate_pltgot() so every PLT jump slot is resolved in AS0 at load time, before any fork(). Lazy PLT binding is not fork-coherent. A forked child running in its own copy-on-write address space that is the FIRST to hit an unresolved PLT slot must run the resolver (elf::object::resolve_pltgot -> object::symbol -> program::lookup) inside that child. Two distinct facets fault there: 1. GOT-slot relocation (fixed in 8d775cc): a demand-paged, relocation -dirtied GOT page not resident at fork is re-read UNRELOCATED from the backing file in the child, so the resolver dereferences a NULL elf::object. 2. Symbol lookup (this change): under heavy fork load (a workload that forks many worker processes which each first-call library functions the parent never touched), a child's lazy resolver RUNS but the dynamic symbol LOOKUP fails for a symbol that IS exported by the kernel and has a normal undefined ref in the app (e.g. sem_wait, preadv, pwritev -- POSIX semaphores and vectored I/O), the "symbol not found" wall. Resolving every slot up front in AS0 removes BOTH by construction: no forked child ever enters the resolver. The resolution also WRITES each GOT slot, so its page becomes anonymous and resident and clone_address_space() COW-shares the RELOCATED contents. bind_now runs in relocate_pltgot() before fix_permissions() write-protects the RELRO segment, so the slot writes never hit a read-only page (the failure mode an earlier post-fork eager-bind attempt hit). This supersedes the mmap_populate-of-writable-segments half of 8d775cc (the bind-now writes already pin the GOT resident) and is complementary to its identity-heap _used_by_resolve_plt_got insert, which now only matters for objects dlopen'd AFTER the first fork. Evidence (conf_fork=1, x64): a temporary trace in resolve_pltgot counted forked-child lazy resolutions across three fork tests: without bind-now 102 / 263 / 681 child resolutions; with bind-now 0 / 0 / 0. New tests tst-fork-symlookup (heavy concurrent fork, wide symbol set incl. sem_wait/preadv/pwritev) and tst-fork-vislookup (PLT-linked dependency resolved first in forked child + a pthread) pass at -smp4 and -smp8, as does tst-fork-pltgot. Boot 10/10. conf_fork=0 elf.o is byte-identical to 8d775cc modulo 11 __LINE__ immediates; zero real code differences.
… 2M fill (fixes the anon-MAP_SHARED populate spin loop) A page provider may deliberately reject a large (2M) fill by returning false from its level-1 map() so the walker uses 4K granularity instead. The fork-coherent shared_anon_page_provider does exactly this: its process-global page registry is keyed on the 4K page VA, so it must be filled one 4K page at a time. populate::page() ignored that false: it always returned true, telling the page-table walker the 2M range was handled. The walker then neither descended to level 0 nor left an intermediate table, so the 2M PTE stayed empty. The faulting instruction retried, faulted on the same address, and looped forever at ~100% CPU -- the endless 2M-populate spin the PostgreSQL postmaster hits the moment it first touches its shared_buffers segment (default shared_memory_type=mmap, an anonymous MAP_SHARED region), before it can reach "ready to accept connections". Fix: when the provider's map() returns false at a large-capable level, populate::page() returns false too, so the walker allocates the level-1 intermediate table and descends to map the 4K pages via the level-0 map(). At the 4K leaf there is no lower level, so a false there is harmless (the walker ignores the return value at level 0). With this, PostgreSQL's default mmap shared memory populates correctly and is routed through the already fork-coherent shared_anon_page_provider, so a forked backend reads the same shared_buffers catalog pages the postmaster filled instead of zero pages. The normal anonymous providers only return false at level 1 on a lost write_pte race (the freshly allocated huge page is dropped); descending to retry at 4K in that case is correct as well. Gated under CONF_fork so the non-fork build is byte-identical.
…ee walls)
Document the resolution of the three walls that blocked stock PostgreSQL from
running a real pgbench workload on OSv under KVM, so the serving path is
reproducible from committed code with no uncommitted deltas and no PostgreSQL
configuration workaround:
W1 (fixed, kernel): the anonymous MAP_SHARED populate spin loop -- the
populate::page() descend fix in core/mmu.cc lets PostgreSQL use its default
shared_memory_type=mmap instead of the sysv workaround.
W2 (already fixed): catalog reads returning zero pages across forked backends
-- solved by the committed shared_anon_page_provider; W1's fix routes the
default mmap shared memory back onto that coherent path (the sysv workaround
had dodged W1 but re-opened W2 by using a different, uncovered shm path).
W3 (downstream of W1, gone): the page_range_allocator::remove fault in the L2
pool fill_thread did not reproduce once the populate spin loop was fixed.
Records the validation (catalog queries, pgbench -i -s50 + -c8 -T30 with 0
failed, boot 5/5) and the exact reproduce steps, including that no config-only
setting is required for the walls (shared_memory_type stays at the mmap
default).
…ty heap (fixes the -smp>=56 boot fault during the shared_buffers populate) The FreeBSD random-harvest queue allocates its ~40 KB interrupt entropy ring with a plain `new ring_t()` at boot. A large allocation of that size is served by mmap and lands in the COW-cloned application mmap slot (VA 0x2000..), not the identity kernel heap. harvest_interrupt_randomness() -> random_harvestq_internal() writes into that ring on EVERY interrupt, from interrupt() with interrupts and preemption off. After stock PostgreSQL (CONF_fork) forks its first process, clone_address_space() COW-write-protects AS0 private-writable anon pages, including the ring page. The very next interrupt-context write to the ring takes a COW write-fault in a non-preemptable IRQ context, tripping assert(sched::preemptable()) in arch/x64/mmu.cc page_fault. The fault surfaces in sched::cpu::idle() (an idle CPU that took the interrupt, running preempt-disabled) and is deterministic at high vCPU count: with more CPUs, more interrupts land on more idle CPUs during the large shared_buffers mmap-populate window, so the COW-protected ring page is reliably written from an interrupt before the COW break happens. The observed ceiling is 64 (OSv already refuses to boot with more than 64 CPUs, max_cpus == 64): clean at <=48, a deterministic preemptable() assert at 56 and 64. gdb ground truth (diagnostic build, -smp 56): cr2=0x2000002023b0 (app mmap slot), error_code=0x3 (present + write == COW write-fault), rflags IF=0 (irq off), preempt_counter=1, faulting rip in random_harvestq_internal writing a ring slot. Fix: allocate the ring under fork_arena::kernel_heap_scope so malloc_large() registers the range fork-shared and clone_address_space() maps it verbatim in every address space (never COW), exactly like the ZFS ARC hash arrays and the other interrupt-reachable kernel structures (timers, net_channel pollers, epoll nodes). The ring is then coherent and writable from every AS and every idle CPU interrupt path. All #if CONF_fork; conf_fork=0 is byte-identical (the #else keeps the original `new ring_t()`). The W1 large-page-descend fix keeps working at <=48; OSv+PG now boots to ready to accept connections 5/5 at -smp 8/32/48/56/64 on default shared_memory_type=mmap (no sysv) and serves pgbench -c8 with 0 failed transactions. -smp 96 is not reachable: OSv caps at 64 CPUs by design. Author: Greg Burd <greg@burd.me>
gburd
force-pushed
the
integ/pg-fork-zfs
branch
from
August 4, 2026 13:21
61678c5 to
8f40c06
Compare
…ling) Every fork-backend page fault on an anon MAP_SHARED segment (e.g. PostgreSQL shared_buffers) resolves the backing page through shared_anon_page_provider, which took a single process-global mutex on shared_anon_registry. Under concurrency all backends serialize on that one lock: at c8+ the lock saturates, tps = concurrency / latency, and the guest sits ~99.9% idle with every backend blocked in lockfree::mutex::lock <- shared_anon_page_provider::shared_page. This is the read-only scaling ceiling that pinned pgbench -S at ~c8 regardless of vCPU count (8/32/48) -- the lock does not scale with cores. Shard the registry into NR_SHARDS (256) buckets, each an independent mutex + map, selected by a cheap mix of the page VA. Faults on distinct pages now take distinct locks and proceed in parallel. Correctness is unchanged: a given VA always hashes to the same shard, preserving the one-shared-frame-per-VA invariant that MAP_SHARED coherence across fork address spaces depends on. (cherry picked from commit 6fe44f0979fe97422362fb6441c6d12320ea8cf5)
…k-free garbage path Under CONF_fork the small-object pool's cross-CPU free path could hand a backing page back to page_pool while the lock-free MPSC garbage queue still held an intrusive free_object::next reference into that page. A concurrent producer's in-flight link store, or the consumer's _poll_list look-ahead, then wrote or read an 8-byte pointer into the recycled page: a page-level use-after-free that corrupted ZFS metadata (~128 KiB aligned) and tripped the boost intrusive assert in page_range_allocator::alloc. The shared kernel small-object pool must stay coherent across every fork address space (it lives in the identity map, PML4 slots 128..511, shared verbatim), so the fix is not to move the pool but to make its cross-CPU page reclaim safe: never return an emptied pool page to page_pool from free_same_cpu. The page stays fully-free on the per-CPU pool _free list, and the genuinely excess empties are released back to page_pool only from collect_garbage, AFTER every incoming garbage sink for the CPU has been drained. At that single-consumer, preempt-locked point no MPSC link references any pool object, so freeing a page cannot leave a dangling intrusive pointer. Retention is bounded (max_retained_empty per CPU per pool) so a bursty free storm cannot pin unbounded memory, which is what turns a naive never-reclaim into an out-of-memory-then-fault. alloc keeps the per-CPU empty tally honest when a page leaves the fully-free state. Gated entirely under CONF_fork: with conf_fork=0 none of this compiles in and the allocator is byte-identical to stock OSv.
…ge-path free_object UAF
The page-path sibling of the pool-reclaim fix. The small-object pool's
lock-free MPSC cross-CPU garbage queue (lockfree::unordered_queue_mpsc) links
freed objects through an intrusive free_object::next stored INSIDE the object's
backing page:
push(): item->next = _head; CAS(_head, item) // 8-byte self-alias store
pop(): _poll_list = r->next // consumer look-ahead
A pool backing page can be returned to page_pool (via pool::flush_empty_pages
-> untracked_free_page, or after it is recycled and re-grabbed by
pool::add_page) during a window in which a producer on another CPU is still
mid-push (the item->next store) or the consumer's _poll_list look-ahead still
dereferences an object in it. The 8-byte next-link store then lands in a page
page_pool has already recycled into a live 4096B object (a ZFS abd read-chunk,
a range_tree btree leaf, or a thread TCB/TLS block), corrupting it. Observed as
the range_tree VERIFY3U(start<end), EIO on vacuum/checkpoint reads (a clobbered
ARC read buffer failing checksum), and the migrate_disable/preemptable
page-fault assert (a clobbered per-thread TLS slot). A poison-on-free /
verify-on-alloc canary caught the stale stores directly at free_object slots.
The pool-reclaim fix deferred the pool's OWN page release to a drained,
preempt-locked point; a concurrent producer push plus the pop() look-ahead
leave a residual window on the direct page path this closes. Rather than chase
every producer, never hand a freed page straight back to page_pool for
immediate reuse: park it in a bounded per-CPU ring (PGQUAR_DEPTH=512 pages,
2 MiB/CPU) and recycle only the oldest after 512 subsequent frees, so any
in-flight MPSC next-link store drains harmlessly onto a page that is not yet a
live object. Bounded so a bursty free storm cannot pin unbounded memory.
Gated entirely under CONF_fork; with conf_fork=0 memory::free_page() is
byte-identical to stock OSv.
Together with the pool-reclaim fix this is what lets stock PostgreSQL sustain
concurrent multi-client writes on OSv: pgbench -c8/-c16/-c32/-c48/-c64 RW all
complete with 0 failed transactions, no EIO, no page-verification failure, and
the pool reads back clean across a reboot with WAL recovery. Without both,
concurrent writes at c>=8 corrupt heap pages (PG page-checksum mismatch) and
ZFS blocks (EIO) on ~128 KiB-aligned runs.
The shared_anon_page_provider resolves the one shared physical page for each VA of a MAP_SHARED|MAP_ANONYMOUS mapping across fork(), keyed by the absolute page VA in a process-global registry. Every fault of such a page in every fork address space consults this registry; for a large shared-anon segment exercised by many forked workers that is a flood of read-only lookups over a table populated by a handful of first-fault inserts. The registry was a 256-way sharded std::unordered_map, each shard behind its own mutex. Under concurrent fork-worker load the shard lock is the measured hot lock on the fault path: the read path took a mutex on every one of millions of lookups. Replace it with an osv::rcu_hashtable (the same read-mostly structure the net-channel classifier uses) plus a single process-global insert mutex. The hit path now takes NO mutex: it runs inside an rcu_read_lock, does a lock-free reader_find(va) and returns the recorded page. Only the rare first-fault insert takes the lock, where a double-checked owner_find under the lock preserves the "one shared physical page per VA" invariant against a concurrent inserting address space. All the existing invariants are unchanged (identity-heap pages and nodes so the table is visible in every fork AS, pte_shared tagging, AS0-reclaim-on-munmap). Generic: this benefits ANY fork-plus-shared-anonymous workload (prefork servers, shared-memory IPC across fork, forked-worker runtimes), not a specific application. CONF_fork-gated; a CONF_fork=0 build is unchanged. Signed-off-by: Greg Burd <greg@burd.me>
A write fault previously took as->vmas_mutex->for_write() -- an AS-wide exclusive lock -- on EVERY write fault, just to call handle_cow_write_fault which walks to the leaf PTE and returns false for a non-COW page. That exclusive hold serializes ALL faults in the address space. For a MAP_SHARED|MAP_ANONYMOUS page across fork the leaf is never a private copy-on-write page: it is either not yet resident (first touch, installed under the for_read fault path) or present as a writable pte_shared entry resolved lock-free through the registry. Such a fault does not need the AS-wide write lock at all. Peek the leaf PTE lock-free before taking the exclusive lock: factor the page-table walk into walk_to_leaf() (shared with handle_cow_write_fault so the walk is written once) and a write_fault_needs_cow_lock() classifier that returns true only for a genuinely present private-COW leaf. Take the AS-wide write lock only then. Shared-anon and first-touch write faults now take the read lock like every other fault and stop serializing behind the writer. The private-COW path is unchanged (it still takes the write lock to make the copy). A toggle env OSV_MMU_COW_PEEK (default on) allows a clean A/B comparison from a single image. Generic: benefits ANY MAP_SHARED-anonymous-across-fork workload, no application awareness. CONF_fork-gated; a CONF_fork=0 build is unchanged. Signed-off-by: Greg Burd <greg@burd.me>
Contributor
Author
Fork-only sched follow-on for the S-split: parked-timer unlink before idle-pullRecording a small The idle-CPU pull work-stealing path ( The fix mirrors Branch: |
Contributor
Author
|
Tracking update: the current authoritative integration tip is now
Two of these coherence fixes are also broken out as standalone master-eligible follow-ons where they stand on their own:
These are recorded here so the completed work is captured in the tracking PR while the trilogy bases (#1455/#1456/#1457) and #1423 await review. No change requested on this PR; it remains DO-NOT-MERGE-as-one-unit. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Draft / tracking — DO NOT MERGE AS ONE UNIT
This is the full PostgreSQL-on-OSv integration branch (
integ/pg-fork-zfs, 67 commits). It is filed as a draft tracking PR to make the completed fork-completeness + ZFS-coherence work visible with its dependency chain, NOT for review as one 67-commit unit.It will be split into the logical follow-on stack (S1..S6 below) once its bases merge. Until then this branch is the integration reference that proves stock, unmodified PostgreSQL boots, forks its aux processes, serves rows, and sustains concurrent load on OSv over OpenZFS-on-NVMe.
Depends on (must merge first)
The fork trilogy (opt-in,
CONF_fork, off by default):process: implement thread-backed fork()/vfork()/execve()/waitpid()mm: per-child copy-on-write address space for fork()(stacked on process: implement thread-backed fork()/vfork()/execve()/waitpid() (opt-in, off by default) #1455)libc: honor POSIX default-ignore disposition for SIGCHLD/SIGURG/SIGWINCHAnd, for the ZFS-coherence + PG-on-ZFS fixes:
zfs: selectable in-kernel ZFS (BSD or OpenZFS 2.4.2) via upstream submodule + patch seriesThis branch already contains the #1423 commits and the fork-trilogy commits inline (it was built on top of them); the split PRs below will be rebased to sit on top of the merged bases so each is independently reviewable.
The intended split (S1..S6), in dependency order
Every fork-side commit on this branch is prefixed by its destination
(
[fork-stack / CONF_fork],fork:,fork arena:,mm/fork:,mmu:,process:), and every ZFS-side commit by[#1423 / OpenZFS]/zfs(openzfs):. The split follows those groupings:S1 — AS-aware mmap allocation path (stacks on #1455 + #1456)
Thread
address_space*through the mmu allocation path (allocate/map_anon/find_hole/evacuate/protect/unmap/mprotect/msync + vma_range_set), defaulting to
the kernel AS so the non-fork path stays byte-identical; give a child AS its own
vma_range_set. Fixes fork-child mmap landing in the global vma_list invisiblyto the child fault handler (the W-mmap wall). Also
fork: preserve file_vma type in clone_address_space.Key commits:
mmu: make the mmap allocation path address-space aware,fork: preserve file_vma type in clone_address_space (fix wild-branch).S2 — Per-child COW arena + same-VA COW stack (stacks on #1456)
Private COW-able heap arena for fork children; COW 2 MB large pages; same-VA COW
stack for deep call chains; lock-free arena allocator; eager arena population;
identity-heap TLS/syscall stacks; intrusive zombie reaping (no alloc on the
lifecycle path).
Key commits: the
mm/fork:trio + the ninefork arena:commits.S3 — fd inheritance across fork (stacks on S1/S2)
Child gets its own reference on inherited fds; keep the shared fd slot when the
owner closes an fd a live child still holds (backend connection-socket wall).
Key commits:
fork: give the child its own reference on inherited fds,fork: keep the shared fd slot when the owner closes an fd a live child inherited.S4 — Per-process signals + timer parking (stacks on #1457)
Per-process signal dispositions + deliver blocked SIGCHLD to the handler
(postmaster PM_RUN wall); park app-thread timers off the per-CPU list across
address spaces (IRQ-context COW wall).
Key commits:
fork: per-process signal dispositions + deliver blocked SIGCHLD,fork: park app-thread timers off the per-CPU list across address spaces.S5 — Cross-AS coherence (kernel structures) (stacks on S1..S4)
Route kernel wait-records / list nodes / latches that a forked backend and AS0
must both see onto the identity kernel heap: rwlock read-waiters, signal-waiters,
rcu_defer wait_records, epoll containers + watcher lists, semaphores, waitqueues,
renamed-dentry d_path,
kill(OSV_PID)routing to AS0 for parallel workers,anonymous MAP_SHARED coherence for
shared_buffers, POSIX-shm/DSM/ramfs/netRX+TX coherence, thread-stack coherence, application_runtime + atfork handlers.
Key commits: the
fork: cross-AS coherence ...set + the[fork-stack / CONF_fork] route ... onto the identity heapset.S6 — ZFS-coherence + PG-on-ZFS durability fixes (stacks on S5 and #1423)
Route ZFS taskqueue + large kmem allocations to the identity linear map/heap so
forked-backend writes don't deadlock; cross-AS coherence for the ZFS
I/O-completion path (bio, zio/SPL heap, libsolaris.so statics); virtio-blk req
coherence. Plus the
[#1423 / OpenZFS]durability patches (0028 vnode v_sizerefresh, 0029 zfs_write_simple/zfs_space for ZIL replay, 0030 single-threaded
inline ARC eviction, taskq_wait quiescence) — these amend #1423, not the fork
stack, per the standing rule.
Split & landing rule (from the program's standing rules)
shipping fork PRs (process: implement thread-backed fork()/vfork()/execve()/waitpid() (opt-in, off by default) #1455/mm: per-child copy-on-write address space for fork() (opt-in, stacked on #1455) #1456/libc: honor POSIX default-ignore disposition for SIGCHLD/SIGURG/SIGWINCH #1457) to strengthen them.
pr/openzfs-draft).(S1..S6 above), gated
CONF_fork; the non-fork OSv path stays byte-identical.Blocks / part of
Part of the PostgreSQL-on-OSv North Star (Milestone 1: stock unmodified PG at
parity with PG-on-Linux over ZFS). This branch is where the M1 walls were
root-caused and fixed; the app side is the osv-apps
postgres18-musldemo(held on this stack landing on master).
Status: WIP / DO-NOT-MERGE as one unit. Will be split into S1..S6 and each
sub-PR opened for review once #1455 / #1456 / #1457 and #1423 merge. Filed as a
draft so the dependency chain and the completed work are visible.