zfs(openzfs): enable a runtime ZFS data pool on a non-ZFS-root image (follow-on to #1423) - #1478
zfs(openzfs): enable a runtime ZFS data pool on a non-ZFS-root image (follow-on to #1423)#1478gburd wants to merge 7 commits into
Conversation
The --preload-zfs-library option loads libsolaris.so into memory but does not create the in-kernel ZFS control device /dev/zfs. For the ZFS builder that was sufficient, but it means an image whose root filesystem is not zfs (for example a ramfs root) cannot create or import a ZFS data pool at runtime: without /dev/zfs the zpool and zfs commands have no channel to the kernel and zpool create fails with /dev/zfs not found. load_zfs_library_and_mount_zfs_root() already calls zfsdev_init() on the fs=zfs path. Do the same on the preload path so a non-zfs-root image can bring up a ZFS data pool. Behavior for the ZFS builder is unchanged (it gets an additional, idempotent control-device init before it runs its own commands).
The ZFS userspace tools (zpool.so, zfs.so) link only against libzfs.so and libc and resolve their nvpair/nvlist/fnvlist symbols against libsolaris.so at runtime. Loading libsolaris with RTLD_LOCAL left those exports out of the global symbol scope, so a later dlopen of zpool.so reported dozens of ignored missing nvpair symbols and then could not create the ZFS control device (/dev/zfs not found), blocking any runtime zpool/zfs command. Load the ZFS kernel library with RTLD_GLOBAL so the tools resolve correctly.
The ZFS userspace tools consult /etc/mnttab; on a non-ZFS root (e.g. ramfs) nothing creates it, so a runtime zpool create/import from a ramfs-root image found no mnttab. Mirror the in-tree zfs bench harness (zfsdev_init + creat /etc/mnttab) in the preload path so a ramfs-root image can bring up a ZFS data pool on a local disk.
dlopen of /usr/lib/fs/libsolaris.so from a bootfs ramfs root fails once the opt_mount block has run its unmount/pivot sequence, which is why the fs=zfs path (which loads libsolaris inside the mount block, before the pivot) worked while a post-mount preload did not. Run the preload before opt_mount so the bootfs path is still resolvable, bringing up the ZFS control device and an empty /etc/mnttab so a ramfs-root image can create a ZFS data pool at runtime.
libsolaris.so's manifest entry lived only on the zfs placeholder module, but the placeholder is replaced by its required provider (open_zfs) during module resolution, so the entry was dropped and libsolaris.so never landed in usr.manifest. For a bootfs-populated fs=ramfs image (whose bootfs is built from usr.manifest) this silently omitted libsolaris.so, so --preload-zfs-library could not dlopen /usr/lib/fs/libsolaris.so and no runtime ZFS data pool could be created. Generate the manifest entry on the open_zfs provider module itself (at import, matching zfs-tools) so it survives resolution and is packed.
…alesce cv_timedwait_hires converted its nanosecond deadline to ddi_get_lbolt ticks and called the tick-granular cv_timedwait. At hz=1000 a tick is 1ms, so the ZIL commit-batch window (zil_commit_waiter_timeout sizes it as a small fraction of the last log-write latency, typically a few hundred microseconds) rounded down to zero. Concurrent fsyncs then never coalesced into one log-write-block plus one device cache flush, so every commit paid its own synchronous flush and sync-commit write throughput did not scale with concurrency. The old code also passed an absolute CALLOUT_FLAG_ABSOLUTE deadline to a relative-timeout tick API. Add openzfs_cv_timedwait_hires (bsd/porting/netport1.cc), which waits the true remaining nanoseconds via the OSv condvar chrono wait and honors the absolute vs relative flag, and route the OpenZFS SPL cv_timedwait_hires to it. The ZIL batch window is now respected and concurrent commits coalesce.
…dline The absolute deadline passed to cv_timedwait_hires is built from gethrtime(), which on OSv is clock_gettime(CLOCK_UPTIME) where CLOCK_UPTIME is #defined to CLOCK_REALTIME, i.e. a wall-clock nanosecond value (order 1.7e18 ns). The previous code subtracted uptime now (order 1e11 ns at boot) from that wall-clock deadline, producing a delay of order 1.7e18 ns (more than fifty years). The ZIL commit-batch timeout timer was therefore armed effectively forever and never fired; a backend that also missed its log-write-completion signal then waited on the condition variable permanently and stalled every later committer behind it. Subtract wall-clock now to match gethrtime()'s clock base, so the timed wait honors the true sub-millisecond commit-batch window. The relative branch already used the wall clock; only the absolute branch was wrong. Comments in netport1.cc, kcondvar.h and the SPL condvar.h that described the deadline as uptime are corrected to wall-clock.
|
Follow-up correctness fix pushed (25bde6b): the absolute-deadline branch of
The fix subtracts wall-clock now to match Diagnosed by reading a wedged guest's stuck backend directly: it was correctly enqueued and unwoken on a ZFS condition variable with an armed timer whose parked deadline was a wall-clock epoch value interpreted as uptime, so the timeout never arrived. |
Follow-on to #1423 (in-kernel OpenZFS). These six changes let an OSv image
whose root filesystem is not ZFS (for example
fs=ramfs) bring up an OpenZFSdata pool on a local disk at runtime and serve an application from it. Each
is small and independent; together they close the gaps that prevented
--preload-zfs-library+ a runtimezpool createfrom working, plus one ZILlatency bug.
1.
loader: init the ZFS control device when preloading libsolaris--preload-zfs-libraryonly loadedlibsolaris.so; it did not create the/dev/zfscontrol device, sozpool/zfscould not talk to the kernel(
/dev/zfs not found). Callzfsdev_init()on the preload path as thefs=zfsroot-mount path already does.2.
loader: load ZFS libsolaris with RTLD_GLOBALThe ZFS userspace tools link only against
libzfs.soand libc and resolvetheir
nvpair/nvlistsymbols againstlibsolaris.soat runtime. Loading itRTLD_LOCALleft those out of the global scope, so a laterdlopenof thetools reported dozens of missing symbols and could not create the control
device. Load with
RTLD_GLOBAL.3.
loader: create /etc/mnttab when preloading ZFSThe tools consult
/etc/mnttab; on a non-ZFS root nothing creates it. Createan empty one on the preload path (mirrors the in-tree ZFS bench harness).
4.
loader: preload the ZFS library before the root-mount blockdlopenof/usr/lib/fs/libsolaris.sofrom a bootfs ramfs root fails once theroot-mount block has run its unmount/pivot sequence, which is why the
fs=zfspath (which loads libsolaris before the pivot) worked while a later preload did
not. Run the preload before the root-mount block.
5.
modules/open_zfs: carry the libsolaris.so manifest entry on the providerThe
libsolaris.somanifest entry lived on thezfsplaceholder module, butthe placeholder is replaced by its provider (
open_zfs) during moduleresolution, so the entry was dropped and
libsolaris.sonever landed inusr.manifest. For a bootfs-populatedfs=ramfsimage this silently omittedit. Generate the entry on the provider module.
6.
zfs(openzfs): nanosecond-precise cv_timedwait_hires so ZIL commits coalescecv_timedwait_hiresrounded its nanosecond deadline toddi_get_lboltticks;at
hz=1000a tick is 1ms, so the ZIL commit-batch window (a few hundredmicroseconds) rounded to zero and concurrent fsyncs never coalesced into one
log write plus one device cache flush. Wait the true remaining nanoseconds via
the OSv condvar chrono wait, honoring the absolute vs relative flag.
Testing
Built (
conf_zfs=openzfs) and booted. With these applied, afs=ramfsOSvimage with
--preload-zfs-librarycreates an OpenZFS pool on a local disk atruntime, and stock PostgreSQL serves with its data directory on ZFS, both under
QEMU and native on the instance.