Skip to content

zfs(openzfs): enable a runtime ZFS data pool on a non-ZFS-root image (follow-on to #1423) - #1478

Open
gburd wants to merge 7 commits into
cloudius-systems:masterfrom
gburd:pr/openzfs-runtime-datapool
Open

zfs(openzfs): enable a runtime ZFS data pool on a non-ZFS-root image (follow-on to #1423)#1478
gburd wants to merge 7 commits into
cloudius-systems:masterfrom
gburd:pr/openzfs-runtime-datapool

Conversation

@gburd

@gburd gburd commented Aug 25, 2026

Copy link
Copy Markdown
Contributor

Follow-on to #1423 (in-kernel OpenZFS). These six changes let an OSv image
whose root filesystem is not ZFS (for example fs=ramfs) bring up an OpenZFS
data pool on a local disk at runtime and serve an application from it. Each
is small and independent; together they close the gaps that prevented
--preload-zfs-library + a runtime zpool create from working, plus one ZIL
latency bug.

1. loader: init the ZFS control device when preloading libsolaris
--preload-zfs-library only loaded libsolaris.so; it did not create the
/dev/zfs control device, so zpool/zfs could not talk to the kernel
(/dev/zfs not found). Call zfsdev_init() on the preload path as the
fs=zfs root-mount path already does.

2. loader: load ZFS libsolaris with RTLD_GLOBAL
The ZFS userspace tools link only against libzfs.so and libc and resolve
their nvpair/nvlist symbols against libsolaris.so at runtime. Loading it
RTLD_LOCAL left those out of the global scope, so a later dlopen of the
tools reported dozens of missing symbols and could not create the control
device. Load with RTLD_GLOBAL.

3. loader: create /etc/mnttab when preloading ZFS
The tools consult /etc/mnttab; on a non-ZFS root nothing creates it. Create
an empty one on the preload path (mirrors the in-tree ZFS bench harness).

4. loader: preload the ZFS library before the root-mount block
dlopen of /usr/lib/fs/libsolaris.so from a bootfs ramfs root fails once the
root-mount block has run its unmount/pivot sequence, which is why the fs=zfs
path (which loads libsolaris before the pivot) worked while a later preload did
not. Run the preload before the root-mount block.

5. modules/open_zfs: carry the libsolaris.so manifest entry on the provider
The libsolaris.so manifest entry lived on the zfs placeholder module, but
the placeholder is replaced by its provider (open_zfs) during module
resolution, so the entry was dropped and libsolaris.so never landed in
usr.manifest. For a bootfs-populated fs=ramfs image this silently omitted
it. Generate the entry on the provider module.

6. zfs(openzfs): nanosecond-precise cv_timedwait_hires so ZIL commits coalesce
cv_timedwait_hires rounded its nanosecond deadline to ddi_get_lbolt ticks;
at hz=1000 a tick is 1ms, so the ZIL commit-batch window (a few hundred
microseconds) rounded to zero and concurrent fsyncs never coalesced into one
log write plus one device cache flush. Wait the true remaining nanoseconds via
the OSv condvar chrono wait, honoring the absolute vs relative flag.

Testing

Built (conf_zfs=openzfs) and booted. With these applied, a fs=ramfs OSv
image with --preload-zfs-library creates an OpenZFS pool on a local disk at
runtime, and stock PostgreSQL serves with its data directory on ZFS, both under
QEMU and native on the instance.

gburd added 7 commits August 25, 2026 03:44
The --preload-zfs-library option loads libsolaris.so into memory but does
not create the in-kernel ZFS control device /dev/zfs.  For the ZFS builder
that was sufficient, but it means an image whose root filesystem is not zfs
(for example a ramfs root) cannot create or import a ZFS data pool at
runtime: without /dev/zfs the zpool and zfs commands have no channel to the
kernel and zpool create fails with /dev/zfs not found.

load_zfs_library_and_mount_zfs_root() already calls zfsdev_init() on the
fs=zfs path.  Do the same on the preload path so a non-zfs-root image can
bring up a ZFS data pool.  Behavior for the ZFS builder is unchanged (it
gets an additional, idempotent control-device init before it runs its
own commands).
The ZFS userspace tools (zpool.so, zfs.so) link only against libzfs.so and libc
and resolve their nvpair/nvlist/fnvlist symbols against libsolaris.so at runtime.
Loading libsolaris with RTLD_LOCAL left those exports out of the global symbol
scope, so a later dlopen of zpool.so reported dozens of ignored missing nvpair
symbols and then could not create the ZFS control device (/dev/zfs not found),
blocking any runtime zpool/zfs command. Load the ZFS kernel library with
RTLD_GLOBAL so the tools resolve correctly.
The ZFS userspace tools consult /etc/mnttab; on a non-ZFS root (e.g. ramfs)
nothing creates it, so a runtime zpool create/import from a ramfs-root image
found no mnttab. Mirror the in-tree zfs bench harness (zfsdev_init + creat
/etc/mnttab) in the preload path so a ramfs-root image can bring up a ZFS data
pool on a local disk.
dlopen of /usr/lib/fs/libsolaris.so from a bootfs ramfs root fails once the
opt_mount block has run its unmount/pivot sequence, which is why the fs=zfs
path (which loads libsolaris inside the mount block, before the pivot) worked
while a post-mount preload did not. Run the preload before opt_mount so the
bootfs path is still resolvable, bringing up the ZFS control device and an
empty /etc/mnttab so a ramfs-root image can create a ZFS data pool at runtime.
libsolaris.so's manifest entry lived only on the zfs placeholder module, but
the placeholder is replaced by its required provider (open_zfs) during module
resolution, so the entry was dropped and libsolaris.so never landed in
usr.manifest. For a bootfs-populated fs=ramfs image (whose bootfs is built from
usr.manifest) this silently omitted libsolaris.so, so --preload-zfs-library
could not dlopen /usr/lib/fs/libsolaris.so and no runtime ZFS data pool could
be created. Generate the manifest entry on the open_zfs provider module itself
(at import, matching zfs-tools) so it survives resolution and is packed.
…alesce

cv_timedwait_hires converted its nanosecond deadline to ddi_get_lbolt ticks and
called the tick-granular cv_timedwait. At hz=1000 a tick is 1ms, so the ZIL
commit-batch window (zil_commit_waiter_timeout sizes it as a small fraction of
the last log-write latency, typically a few hundred microseconds) rounded down
to zero. Concurrent fsyncs then never coalesced into one log-write-block plus
one device cache flush, so every commit paid its own synchronous flush and
sync-commit write throughput did not scale with concurrency. The old code also
passed an absolute CALLOUT_FLAG_ABSOLUTE deadline to a relative-timeout tick
API.

Add openzfs_cv_timedwait_hires (bsd/porting/netport1.cc), which waits the true
remaining nanoseconds via the OSv condvar chrono wait and honors the absolute
vs relative flag, and route the OpenZFS SPL cv_timedwait_hires to it. The ZIL
batch window is now respected and concurrent commits coalesce.
…dline

The absolute deadline passed to cv_timedwait_hires is built from gethrtime(),
which on OSv is clock_gettime(CLOCK_UPTIME) where CLOCK_UPTIME is #defined to
CLOCK_REALTIME, i.e. a wall-clock nanosecond value (order 1.7e18 ns). The
previous code subtracted uptime now (order 1e11 ns at boot) from that
wall-clock deadline, producing a delay of order 1.7e18 ns (more than fifty
years). The ZIL commit-batch timeout timer was therefore armed effectively
forever and never fired; a backend that also missed its log-write-completion
signal then waited on the condition variable permanently and stalled every
later committer behind it.

Subtract wall-clock now to match gethrtime()'s clock base, so the timed wait
honors the true sub-millisecond commit-batch window. The relative branch
already used the wall clock; only the absolute branch was wrong. Comments in
netport1.cc, kcondvar.h and the SPL condvar.h that described the deadline as
uptime are corrected to wall-clock.
@gburd

gburd commented Aug 25, 2026

Copy link
Copy Markdown
Contributor Author

Follow-up correctness fix pushed (25bde6b): the absolute-deadline branch of cv_timedwait_hires was using the wrong clock base.

gethrtime() on OSv is clock_gettime(CLOCK_UPTIME), and CLOCK_UPTIME is #defined to CLOCK_REALTIME (see bsd/sys/cddl/compat/opensolaris/sys/time.h), so an absolute gethrtime() deadline is a wall-clock nanosecond value (order 1.7e18 ns). The previous code subtracted uptime-now (order 1e11 ns at boot) from that wall-clock deadline, yielding a delay of order 1.7e18 ns, roughly fifty years. The ZIL commit-batch timeout timer was armed effectively forever and never fired. A backend that also missed its log-write-completion signal then waited on the condition variable permanently and stalled every later committer behind it, wedging synchronous-commit writes under concurrency.

The fix subtracts wall-clock now to match gethrtime()'s clock base. The relative branch already used the wall clock; only the absolute branch was wrong. Comments describing the deadline as uptime are corrected.

Diagnosed by reading a wedged guest's stuck backend directly: it was correctly enqueued and unwoken on a ZFS condition variable with an armed timer whose parked deadline was a wall-clock epoch value interpreted as uptime, so the timeout never arrived.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant