Skip to content

mm: resolve fork COW write faults lock-free when nested (fork-stack follow-on to #1456) - #1475

Draft
gburd wants to merge 7 commits into
cloudius-systems:masterfrom
gburd:pr/fork-cow-nested-lockfree
Draft

mm: resolve fork COW write faults lock-free when nested (fork-stack follow-on to #1456)#1475
gburd wants to merge 7 commits into
cloudius-systems:masterfrom
gburd:pr/fork-cow-nested-lockfree

Conversation

@gburd

@gburd gburd commented Aug 17, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

DRAFT / fork-stack follow-on -- not a standalone master change.

This is a one-commit follow-on to the opt-in fork() copy-on-write work
(#1456), and belongs on that stack, tracked under #1458. It is not
proposable against stock master on its own: the code it touches
(handle_cow_write_fault and the nested COW branch of vm_fault) only
exists under #if CONF_fork, which lands with #1455/#1456. On a
conf_fork=0 build the change compiles out entirely and the binary is
byte-identical.

The reviewable unit here is the single top commit mm: resolve fork COW write faults lock-free when nested ... touching core/mmu.cc only. The
other commits shown in the diff are the #1456 COW base this stacks on; they
appear because master does not yet carry that base. Review just the top
commit.

What it fixes

A copy-on-write write fault can be taken while another exception is already
in flight -- e.g. a device/timer IRQ lands a fault while a COW fault is
already being serviced during a fork() -- driving the fault handler to
exception_depth >= 2. At that depth no reschedule is allowed:
reschedule_from_interrupt asserts exception_depth <= 1 (the per-CPU
nested-exception stacks are finite).

The normal COW path reschedules in two places, either of which trips that
assert deep in the fork path under concurrency:

  • it takes the AS-wide vmas_mutex->for_write() -- a contended rwlock that
    blocks (and therefore reschedules), and
  • handle_cow_write_fault calls flush_tlb_all(), which takes
    tlb_flush_mutex and wait_until()s every other CPU (also a reschedule).

The change

When the COW write fault is itself nested (exception_depth >= 2), resolve
it lock-free: handle_cow_write_fault already walks only this address
space's own page tables and installs the private page with a single PTE
write to this AS, so a local TLB flush suffices (the page is private to
this AS; other CPUs run a different CR3 and reload on switch-in). Nothing
blocks, so no reschedule can happen at exception_depth >= 2. A concurrent
COW copy of the same page by another CPU at most duplicates a private page
(the loser leaks one page -- rare, only in this nested race, and
correctness-preserving).

The non-nested (exception_depth <= 1) path is byte-for-byte unchanged.

Why it matters

Beyond the crash-safety (it removes an assert that fires deep in the fork
path under concurrent load), it also removes a latency serialization for
forked-process servers: a backend blocking on the AS-wide vmas_mutex and a
global cross-CPU TLB flush on the nested COW write-fault path serializes
concurrent forked-process workloads. Resolving the nested fault lock-free
with a local flush lets those backends make progress independently.

Scope / stacking

Draft while the fork stack is gated on #1455 -- opened so the fix is banked
reviewably on the right stack, not to request near-term merge.

gburd added 7 commits July 31, 2026 09:18
@gburd
process: implement thread-backed fork()/vfork()/execve()/waitpid() (o…
3991815 
…pt-in)

OSv is a single-address-space unikernel, so a literal copy-on-write fork() is
not the model.  This adds a thread-backed fork() emulation that covers the
useful, compatible subset of fork semantics, gated entirely behind a new
CONFIG_fork kconfig option that defaults to OFF - so a default OSv build is
unchanged (none of this code is compiled in and fork()/vfork() return ENOSYS as
before).

With CONFIG_fork enabled:
- fork()/vfork() create a child OSv thread that resumes in fork()'s caller and
  returns 0 in the child / the child pid in the parent (the classic twin
  return).  The child runs on a private copy of the parent's user stack, so
  parent and child have independent locals after the return; it gets its own
  fresh OSv per-thread TLS block (own errno etc).
  Implemented in libc/process/fork.cc + arch/{x64,aarch64}/fork.cc.
- execve() launches the target as a fresh OSv application (its own ELF
  namespace) and does not return, making fork()+exec() work.
- waitpid()/wait4()/wait() reap a child's exit status via a pid->child
  registry; SIGCHLD is raised to the parent on child exit; SIGCHLD/SIGURG/
  SIGWINCH now correctly default to ignore (not poweroff).
- exit()/_exit() in a fork child ends only that child, not the whole unikernel.
- pthread_atfork prepare/parent/child handlers are now actually run around
  fork() (they were a no-op stub) - glibc/musl register these internally.
- sys_clone() routes the non-CLONE_THREAD (fork) case here when enabled.

Validated: tst-fork passes 10/10 on both x86-64 and aarch64 (twin return,
private-stack isolation, fork+exec, vfork, waitpid reaping).

Documented limitations (documentation/fork.md): the child shares the parent's
heap/globals (no per-process memory isolation in one address space - a follow-up
adds per-child copy-on-write address spaces behind the same flag); deep-call-
chain child unwind and fork-as-memory-snapshot (Redis BGSAVE) are not carried by
this base; execve()'s new-ELF-namespace path has a separate pre-existing fault.

Copyright (C) 2026 Greg Burd
@gburd
process: fix execve()/fork() new-namespace exec hanging OSv shutdown
0af931e 
execve() (CONF_fork) launches the target as a fresh OSv application in its own
ELF namespace via application::run(new_program=true) and, matching Linux, does
not return to the caller.  A successful exec DID launch the program (the
elf::program new-namespace construction path is fine), but after the exec'd
program finished the unikernel would hang at shutdown instead of powering off:
the loader's application::join() blocked forever on the top-level app's
_terminated flag.

Root cause is the fork child thread's lifecycle, not execve or elf::program.
fork_thread() (arch/{x64,aarch64}/fork.cc) creates the child as a normal
*attached* sched::thread.  At construction the thread captures a shared_ptr to
the current application's application_runtime (sched.cc sets _app_runtime =
app->runtime()).  Nothing ever join()s the fork child -- the parent reaps it
through the fork pid registry / waitpid(), not sched::thread::join() -- so the
thread object is never destroyed and its _app_runtime shared_ptr is never
released.  With that reference outstanding the application_runtime's use count
never reaches zero, ~application_runtime never runs, the app's _terminated is
never set, and application::join() waits forever.  (This bit every fork(), and
was most visible after fork()+execve() where the child adopts the caller app's
runtime.)

Fix: create the fork child detached and dispose it in its cleanup.
- arch/{x64,aarch64}/fork.cc: mark the child attr().detached(), so on
  completion it is handed to the thread reaper (which runs its set_cleanup()),
  rather than sitting forever waiting to be joined.
- libc/process/fork.cc: the child's cleanup now also calls
  sched::thread::dispose(child) after the existing bookkeeping (record exit
  status if it fell off the end, free the copied user stack).  Disposing the
  thread releases its _app_runtime reference, letting the owning application's
  runtime drop to zero so join() completes and OSv powers off.  This mirrors
  the default detached-thread cleanup ([this]{ dispose(this); }).

Both files are compiled only under CONF_fork, so default OSv builds are
unchanged.

Tests: tst-fork test 2 now execs a real payload (/tests/payload-exit7.so, which
prints a marker and exit(7)s) instead of the previous _exit(7) sentinel that
masked whether execve actually launched anything; a return from execve() is now
treated as a failure.  Added tst-execve.so (fork+execve launches the payload and
its exit code is reaped; missing path returns -1/ENOENT) and payload-exit7.so.

Validated on x86-64 (KVM disk boot): tst-fork 10/10 and tst-execve 3/3 pass with
the real exec payload and OSv shuts down cleanly (5/5 repeat runs, no hang).

Copyright (C) 2026 Greg Burd
@gburd
mm: per-child copy-on-write address space for fork() (opt-in, stacked)
39341d1 
Follow-up to the base fork() PR: gives a forked child its own address space
with copy-on-write of private mappings, so a forked child has real memory
isolation like Linux fork() while MAP_SHARED / shm stays shared.  Also gated
behind CONFIG_fork (default off): with it disabled none of this is compiled and
OSv's single-address-space model and context-switch path are unchanged.

With CONFIG_fork enabled:
- mmu::address_space object (page-table root + vma_list); the previous global
  becomes 'address space 0' (kernel + init app).
- Per-thread current address space; the context switch loads the target CR3
  only when the address space differs (a no-op for AS0-only workloads).  Kernel
  PML4 entries are shared across all address spaces so OSv code + the kernel
  heap work identically after a switch.
- fork() clones the parent's vmas into a child address space: PRIVATE writable
  mappings are write-protected in both and copied on the first write (reusing
  OSv's existing COW fault machinery); MAP_SHARED / shm map the same physical
  pages (truly shared).  execve() returns the thread to AS0.
- Lock/condvar wait_records for fork-child (non-AS0) threads are allocated from
  the AS-shared kernel heap instead of the thread stack, so a wait_record queued
  on a shared kernel mutex resolves to the same physical page for any waker
  across address spaces (kernel-stack coherence).

Validated: tst-fork-cow proves a forked child's private memory stays private
while MAP_SHARED stays shared; tst-fork stays 10/10.

Known limitation (documented): the child's stack is still relocated+copied
rather than same-VA COW, so a child that unwinds a very deep call chain (e.g. a
multi-process server forking backends) can still hit a stack-fidelity issue
(tst-fork-deep); the same-VA stack fix is a further follow-up.  This PR delivers
memory-isolated fork for the common cases with COW proven.

Copyright (C) 2026 Greg Burd
@gburd
mm/fork: same-VA COW stack for deep call chains (opt-in)
cf3d77d 
Give a forked child its parent's EXACT user stack -- same virtual
addresses, private physical pages -- instead of a relocated copy with a
biased SP.  A relocated+biased stack leaves every saved rbp, return
address and &local in the copied frames pointing at the parent's stack
(off by the bias), so a deep unwind in the child dereferences the parent
and faults.  tst-fork-deep (fork 12 frames deep, child unwinds all the
way up and _exit()s a checked value) is the test that catches this; it
now passes.  Gated behind CONF_fork (default n); conf_fork=0 is unchanged
and builds/boots clean (fork code fully excluded, tst-mmap passes).

Four coupled changes make same-VA correct, layered on the Stage 2 COW
address space and the fork-child lifecycle fix (detached child +
dispose(child) in the reaper):

1. core/mmu.cc: clone_address_space() PRIVATIZES the forking thread's
   live stack VA range into the child AS -- fresh private pages that
   byte-copy the parent's stack, mapped writable (not COW: OSv runs
   kernel code with irqs off on the app stack, a COW fault there is
   illegal) at the SAME VA.  The parent's PTEs are untouched.

2. arch/x64/fork.cc + libc/process/fork.cc + include/osv/fork.hh: the
   child resumes with the caller's FULL callee-saved register context
   (rbx, rbp, r12-r15, rsp, rip), captured at fork() entry into
   osv::fork_resume_ctx and restored by the child trampoline off a
   scratch base register (so the restore never clobbers its own base).
   fork() jmps past its own epilogue, so without this the caller would
   resume with fork()'s internal register values -- a deep chain that
   keeps an accumulator in a callee-saved reg across fork() computes
   garbage.

3. arch/x64/arch-switch.hh: defer the fork CR3 switch to EXACTLY the
   rsp/rbp swap inside switch_to()'s asm.  Loading the incoming AS's
   CR3 early, while still executing on the outgoing thread's stack (the
   fpu-cw/mxcsr scratch saves), resolved those shared-VA stack writes
   through the wrong address space and clobbered a blocked thread's live
   stack (root-caused via hardware watchpoint: switch_to wrote garbage
   into a blocked parent's on-stack user_mutex slot in AS0).

4. core/lfmutex.cc + core/mmu.cc: a wait_record on a shared kernel
   condvar/mutex must live in the identity-mapped kernel heap (coherent
   in every AS) whenever a cross-address-space wake is possible.  Extend
   the Option-A rule from 'current thread is a fork child' to also
   cover 'any child address space is live' (live_child_address_spaces
   counter), so the parent in AS0 -- woken by a child that walks the
   queue through its own page tables -- is safe too.

libc/process/execve.cc: on a fork child, move the thread onto its own
kernel stack before switching to AS0 and running the target, so
reloading CR3 to AS0 cannot alias the same-VA app stack onto the
parent's physical page (which would corrupt a blocked parent).  The
child address space is destroyed HERE, on the kernel stack, exactly once:
the reap-time cleanup (libc/process/fork.cc) then sees the thread's AS is
AS0 (!= child_as) and skips its own destroy, so destroy_address_space()
runs once whether the child exits normally (reaper destroys) or execs
(execve destroys) -- no leak, no double-free.  A pre-exec existence check
keeps a failed exec returning to the caller.

Validated on x86-64 (KVM): tst-fork 10/10, tst-fork-cow 6/6,
tst-fork-deep 0 failures, tst-execve 3/3 with clean shutdown (no hang, no
double-free at reap), on smp=1 and smp=4, 5/5 repeat runs; conf_fork=0
builds and boots clean (fork excluded, tst-mmap passes).
@gburd
mm/fork: copy-on-write 2 MB large pages on fork (opt-in)
95a650c 
The per-child COW address space (clone_pt_level<1>) shared any 2 MB large
page verbatim between parent and child instead of copy-on-writing it:

    if (ppte.large()) { child_pt[i] = ppte; continue; } // 2MB: share as-is

so a forked child writing a private large mapping scribbled the parent(s)
too.  This hit large malloc (>= 2 MB, mmap-backed), large MAP_ANON regions,
and any 2 MB-backed private writable mapping in the application slots.

Fix: for a WRITABLE large PD entry, split it to 4 K in the parent in place
(the existing split_large_page primitive) and fall through to the normal 4 K
clone path, which already makes the correct per-page decision -- private
writable -> COW, MAP_SHARED -> stays shared, read-only -> shared as-is.  A
read-only large page can never diverge, so it is still shared verbatim.  This
reuses the whole 4 K COW fault machinery (clone_pt_level0 +
handle_cow_write_fault); the only cost is one 4 K page table per split large
page, acceptable under the opt-in CONF_fork.

Verified on x86-64 (KVM microvm):
  - a standalone musl-PIE fork test now isolates a private >2 MB malloc across
    fork (child write does not change the parent; parent write does not change
    the child), while a MAP_SHARED region stays mutually visible;
  - tst-fork 10/10, tst-fork-cow 6/6, tst-fork-deep and tst-execve all pass
    (no regressions);
  - conf_fork=0 builds clean with the whole fork path (this change included)
    compiled out (clone_address_space absent from loader.elf).

Note: this fixes large-page (application-slot) COW.  OSv's small-object
malloc heap lives in the shared kernel identity map (mem_area::mempool), which
fork() shares by design; isolating that is a separate change.

All gated behind CONF_fork (default off).

Copyright (C) 2026 Greg Burd
@gburd
mm: preserve file_vma type when cloning a child address space on fork
e23e282 
clone_address_space() rebuilt EVERY child VMA as an anon_vma, so a fork
child's file-backed mappings (file_vma -- e.g. the ELF loader's file-backed
.text, mapped via map_file) became anonymous in the child.  A demand fault on
a child file-backed page that the parent had not already resident-loaded then
dispatched to the base zero-fill path instead of file_vma::fault reading the
file: the child read (or executed) zeros.  This is the "wild-branch" bug that
blocked file-backed fork workloads such as PostgreSQL, whose checkpointer child
faults large cold .text regions the postmaster never touched.

Preserve each parent VMA's dynamic type across the clone:

  - In the VMA-clone loop (under the parent's vmas_mutex), dynamic_cast each
    vma to file_vma.  For a file_vma, rebuild the child's copy via the file's
    own mmap() -- the SAME call map_file and file_vma::split use -- so the
    child gets the correct page_allocator (map_file_page_read for MAP_PRIVATE,
    map_file_page_mmap for a cached/shared mapping) and its demand faults read
    the file.  The new file_vma holds its own fileref.  Anon vmas still clone
    as anon_vma.  file::mmap()/default_file_mmap()/map_file_mmap() are pure
    allocation (no locks, no I/O), matching the anon_vma allocation this loop
    already performs under the lock.

  - In destroy_address_space(), dispose the child's owned vma objects before
    freeing the address space.  ~file_vma releases the fileref this child took
    (and deletes its page_allocator), closing the file-reference leak the type
    preservation would otherwise introduce -- and also the pre-existing
    anon_vma + edge-marker leak.  ~vma touches no global list, so disposing
    here is safe.

The whole change is inside #if CONF_fork, so a conf_fork=0 build is
byte-identical.

Adds tests/tst-fork-file-mmap.cc (gated via the fork test group): a parent
mmaps a read-only ROFS file MAP_PRIVATE without faulting a mid-file page,
forks, and the child faults that page first -- it must read the real file
bytes, not zeros.  Reproduces the bug on the unfixed tree (child reads 0x00)
and passes with this fix (child reads the real byte).

Signed-off-by: Greg Burd <greg@burd.me>
@gburd
mm: resolve fork COW write faults lock-free when nested to avoid resc…
0b72ee0 
…hedule at exception_depth >= 2

A copy-on-write write fault taken while another exception is already in
flight (a real IRQ landing a fault during a COW fork, driving the fault
handler to exception_depth >= 2) must not reschedule: the per-CPU
nested-exception stacks are finite and reschedule_from_interrupt
asserts(exception_depth <= 1).  The normal COW path both acquires the
AS-wide vmas_mutex->for_write() (a contended rwlock that blocks) and, in
handle_cow_write_fault, calls flush_tlb_all() (takes tlb_flush_mutex and
wait_until()s every CPU) -- either reschedules, tripping the assert deep
in the fork path under concurrent load.

When nested that deep, resolve the fault lock-free: handle_cow_write_fault
already walks only this address space's own page tables and installs the
private page with a single PTE write to this AS, so a LOCAL TLB flush
suffices (the page is private to this AS; other CPUs run a different CR3
and reload on switch-in).  Nothing blocks, so no reschedule can happen at
exception_depth >= 2.  A concurrent COW copy of the same page by another
CPU at most duplicates a private page (the loser leaks one page -- rare,
only in this nested race, and correctness-preserving).

The non-nested (exception_depth <= 1) path is byte-for-byte unchanged, and
the whole change is under #if CONF_fork so conf_fork=0 builds are
unaffected.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@gburd