ABI: Restore linear ABI 43 history - #1275
Conversation
This reverts commit bd28679.
Serialize supported tagged catch state per activation and reject non-reconstructible reference and table state. Require the ABI 43 safety capability across build and host launch boundaries. (cherry picked from commit 4efcab8)
Give Node and browser process Workers the same versioned reference graph, external-reference authority, imported state, table replicas, exceptions, and replay-gate owners. Recreate main and side-module Store-local identities before continuation rewind. Release temporary roots on completion, abort, exec, and teardown. Stage dynamic loading as host-only prepare/next operations followed by ordinary Wasm initializers. Serialize pthread loader ownership, replay exact module layouts, and defer signal delivery until libc regains the guest boundary. (cherry picked from commit 95fad1e)
Require ABI 43 activation-state-safe capability, exact role metadata, zero native start sections, and staged-loader inventory across package receipts, indexes, sidecars, Homebrew validation, and shell guards. Build valid compiler-EH fixtures normally instead of routing them to an unsupported bucket. Select the declared LLVM archiver on Darwin so final-key source rebuilds remain reproducible. Treat executable kernel.kernel_fork and side-module env.fork as fork-entry evidence in the common binary inventory. Keep the current Homebrew program projection and standalone resolver unchanged; regenerate both once from the coherent batch tip. (cherry picked from commit e0fe315)
Move host-to-kernel structured transfers into bounded, ABI-described scratch regions owned by the kernel. Replace guest-pointer-dependent marshalling for process, IPC, socket, spawn, and signal state with checked wire formats and exact pointer-width layouts. Generate matching Rust, C, and TypeScript metadata, enforce allocation and copy-back bounds, and cover native, wasm32, wasm64, Node, and browser paths. This is the foundational ABI 43 change; later commits close additional ownership and publication seams. (cherry picked from commit 2a99670)
Package cached toolchain sysroots without embedding the source worktree path. Reconstruct the expected layout after cache restore so the action works in a different checkout and on a different runner. (cherry picked from commit 2950040)
Declare and stage the wasm64 examples needed by browser tests instead of assuming that a previous local build left them in place. Teach CI scope detection, workspace packing, and browser fetch tests to enforce the fixture manifest. (cherry picked from commit bd2f8a2)
Finish moving blocked retries, VFS operations, IPC, sockets, process state, and scalar results behind kernel-owned scratch and entry gates. Remove remaining host dependence on mutable guest pointer layouts. Apply the same ownership rules in Node and browser workers. Add focused tests for append and offset behavior, environment transactions, worker lifecycle, and fail-stop scratch violations, then refresh the ABI 43 snapshot and generated bindings. (cherry picked from commit 9dac1cf)
Publish each kernel scratch region and its generated contract as one atomic ABI state. Fail closed when the host observes a missing, stale, or partially initialized region rather than continuing with mixed metadata. Refresh the ABI projection and exercise publication, reusable kernel stacks, and fail-stop validation in the shared Node and browser paths. (cherry picked from commit d17028e)
Honor the Node bridge endpoint-ownership flags after FIN or cleanup. A close may synchronously reclaim the global pipe slot, so later pumps must neither query nor notify its reusable numeric index. (cherry picked from commit 2266566)
Give Node conformance runs an explicit, session-owned VFS root and mount lifecycle. Prevent tests from depending on or mutating ambient host storage, and make teardown retire the same mount state that was admitted. Route the libc, POSIX, and Sortix runners through that production host path and cover mount creation, export, and cleanup. (cherry picked from commit 9713f16)
Reject VFS seed entries whose normalized paths shadow an existing session mount or another seed. This keeps the admitted image authoritative and prevents path ordering from silently changing machine state. (cherry picked from commit 9e78b6d)
Snapshot executable bytes when the resolver admits a program instead of rereading a mutable host file during worker launch. The process now runs the exact image that passed ABI and artifact validation. Cover package resolution and rootfs export while preserving the same behavior in the shared Node and browser launch machinery. (cherry picked from commit 4255846)
Apply retired-memory debt admission synchronously before ordinary fork creates or copies a child WebAssembly.Memory. Return EAGAIN when the bounded debt limit is saturated, without yielding across the parent snapshot or allocating another complete address space. Keep ordinary fork semantics independent from vfork and prove that the rejected path performs no child allocation or copy. (cherry picked from commit 077c844)
Add explicit ownership tokens for host workers that intentionally share one process-memory backing. Count the backing once, retire it only after the final alias crosses its quiescence fence, and propagate forced termination until final release. This ownership mechanism does not reinterpret ordinary fork: only callers with an explicit shared-memory transaction may retain an alias. (cherry picked from commit 3fbb512)
Drive ordinary-fork admission through the production kernel entry gate instead of constructing a partial worker object. Verify that a retired-memory EAGAIN removes the provisional child and completes the parent mailbox with the truthful error. (cherry picked from commit 2de2bb8)
Allow a separately launched child worker to borrow the parent process continuation while keeping worker-local replay control state independent. Rebuild main-module and active side-module references from admitted bytes without handing the child authority over parent continuation storage. Gate completion through an explicit lifetime coordinator and cover main, dynamic-link, and active-side replay in Node, Chromium, Firefox, and WebKit fixtures. (cherry picked from commit d18c1e5)
Document the fork memory root cause, the shared-memory vfork design, ordinary-fork admission, rejected alternatives, ABI implications, and the evidence still required for release. Add a component resident-set-size harness that separates worker, module, shared-memory, full-clone, and sparse-clone costs. Its results are scoped to those components and do not claim Homebrew application performance. (cherry picked from commit d43c1cc)
Binaryen --fpcast-emu rewrites indirectly called functions to a uniform i64 trampoline signature. Host pthread entry calls were still passing a plain JavaScript number, which traps when the wrapper expects BigInt arguments. Build the call arguments from the WebAssembly function parameter count: preserve the plain pointer ABI for ordinary entries and use a BigInt pointer plus zero-filled i64 slots for fpcast trampolines. ABI 43 forward-port: fork-from-thread children now enter through the exported wpk_fork_resume_thread helper. Retain that plain replay ABI and apply fpcast-expanded arguments only where the host calls the table entry directly. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> (cherry picked from commit a2b70da) (cherry picked from commit ca584ca)
Character-special mode is broader than terminal identity. Kandelo encoded host terminal stdio and virtual devices such as /dev/null, framebuffer, audio, and DRM as CharDevice, so terminal probes could mistake any virtual character device for an interactive terminal. Derive terminal identity from the open file description. Dedicated PTY master and slave types are terminals; legacy host stdio is a terminal only when its stable canonical path and host handle agree. Reuse that result for isatty, termios, ioctl namespace gating, and fpathconf. The ABI 43 forward-port retains the current exact 60-byte termios layout. This changes no syscall number, marshalling rule, exported signature, structure layout, generated binding, or VFS ABI metadata. Validation: - native kernel unit suite: 1,491 passed - ABI snapshot and generated bindings check - git diff --check (cherry picked from commit 3b470d3) (cherry picked from commit 47b1d02)
Windows does not expose a POSIX permission model through Node. Native entries carry no owner/group/other split or directory search bit, so a guest that drops privileges cannot traverse or write the host-backed sandbox it was given. On Windows only, represent writable host directories as 0777, read-only directories as 0555, writable files as 0666, and read-only files as 0444. Preserve native type bits and let guest chmod and chown metadata override the synthesized permissions. POSIX hosts keep their native modes unchanged. The ABI 43 forward-port performs synthesis only after exact conversion of the native BigInt mode. Existing EOVERFLOW checks for modes, links, sizes, and timestamps remain intact. Validation: - host declaration generation and typechecking - native-metadata and Node uid/gid suites: 15 passed - Windows host execution was not available on this macOS worktree - git diff --check Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> (cherry picked from commit aa12186) (cherry picked from commit 3ce6274)
A new git worktree (and a fresh checkout) does not inherit git submodules, a musl sysroot, node_modules, or fetched test binaries, so Vitest and the conformance/browser suites cannot run until those are built or fetched. Agents were treating this as "cannot validate" rather than a setup step. Document the exact sequence in validation.md and build-docs-and-prs.md: submodule init (with the stray-libc/musl recovery), build-musl.sh (sysroot), build.sh (kernel wasm → local-binaries/kernel.wasm + rootfs), root + host `npm ci` (root provides tsx for the conformance runners), and fetch-binaries.sh. Add the explicit instruction not to report "I can't run Vitest/conformance/browser" because a fresh worktree lacks artifacts — build or fetch them and report the real result, or name the exact step that failed. Verified by running the sequence end-to-end in a fresh worktree: kernel wasm + sysroot + rootfs build, `vitest` (778 pass; the only failures were a missing fetched `programs/wasm64/hello64.wasm`), and Sortix conformance (10/10) all ran. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> (cherry picked from commit a3739a2) This conceptual commit combines recovery commits cda80003f and 050a7875e. (cherry picked from commit e4ffac7)
Make Area: Purpose the repository convention for PR titles and commit subjects. Add contract-oriented examples and guidance for choosing one primary prefix for cross-cutting changes. Validation: - VitePress documentation build - git diff --check (cherry picked from commit 60fb395) (cherry picked from commit 60df212)
Chromium-based user agents also contain AppleWebKit. Treating that token as sufficient enabled WebKit-only post-destroy memory handling in Chrome, Edge, Opera, and Chromium on iOS. Forward-port the predicate into the current shared kernel-owned boot helper and keep a pure user-agent classifier for regression coverage. Exclude Chromium, Chrome on iOS, Edge, Opera, Firefox, and Firefox on iOS while retaining Safari and WebKit. Validation: - browser engine detection test: 1 passed - git diff --check (cherry picked from commit 4cbaf2b) (cherry picked from commit 7846eb3)
Mutate the sorted mapping vector in place instead of draining and rebuilding it for every munmap call. Preserve the current ABI 43 page-rounding behavior and retain the vector allocation when entries are removed. Keep the original kernel-trap syscall-ring diagnostic while adapting it to the current fatal-entry and blocking-retry cleanup paths. Validated with: scripts/dev-shell.sh -- scripts/ci-run-test-suite.sh cargo-kernel scripts/dev-shell.sh -- npm --prefix host run typecheck End-to-end performance has not been measured yet; that remains part of the batched Node and browser validation. (cherry picked from commit 18a5aa3) (cherry picked from commit 56f0753)
POSIX loopback is machine-scoped. Kandelo previously searched only the sender process socket arena, so an unconnected sendto could report success while dropping the datagram and a connected socket could retain a false ECONNREFUSED. Capture an owned route after the process-local send, release the direct process-table borrows, and then deliver to one accepting foreign endpoint. This preserves sender metadata, avoids duplicate delivery across inherited or SO_REUSEADDR bindings, and clears the provisional connected-socket error only after successful foreign delivery. This changes no ABI exports, imports, or repr(C) layouts. Validated with: scripts/dev-shell.sh -- scripts/ci-run-test-suite.sh cargo-kernel scripts/dev-shell.sh -- bash scripts/check-abi-version.sh scripts/dev-shell.sh -- npm run docs:build (cherry picked from commit c413bd8) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> (cherry picked from commit 36580fb)
Following stat and fstatat calls on /dev/fd/N and the standard-stream aliases now return the referenced descriptor’s authoritative metadata. No-follow calls report stable devfs symlink metadata, readlink exposes the alias target, and O_NOFOLLOW rejects the final symlink. Keep devfs directory, getdents, and lstat device/inode metadata coherent. Add a guest regression exercised through NodeKernelHost and BrowserKernel. The browser test uses the current minimal runner so it does not activate unrelated shell-package generations. This fixes GNU coreutils mistaking /dev/null for stdin without changing the ABI. (cherry picked from commit 35f0455) (cherry picked from commit 3004c2b)
Place Kandelo syscall glue and the C runtime startup object before user link inputs, preserve the caller source, archive, and library order, and keep the final musl archive last. This prevents an explicit -lc from resolving syscall definitions before the platform overrides and avoids scanning dependency libraries before their consumers. Adapt the original change to the current prepared-linker and reproducible path logic. The Kandelo-native driver already uses the same order. (cherry picked from commit 915a84b) (cherry picked from commit 9ef2d3e)
Open the replacement iterator before retiring the active stream, and keep the old state authoritative if opening or closing the replacement sequence fails. Add kernel failure coverage and shared host-bridge coverage for throwing close calls and numeric handle reuse. Adapt the host tests to the current Rust-lent destination capability instead of bypassing private state. (cherry picked from commit e96c381) (cherry picked from commit 052be9c)
This concept commit consolidates its complete contiguous change from the original ABI 43 batch. Original-commit: b385984 Original-commit: 156ef54 Original-commit: 5b8e504 Original-commit: 90be7d6 Original-commit: f433d14 Original-commit: 63d3728 Original-commit: 1e2b243 Original-commit: 6b04112 Original-commit: 7a80d12
(cherry picked from commit c66ccdb)
(cherry picked from commit 66468b6)
(cherry picked from commit d6b3814)
(cherry picked from commit 1aa17208e4b750559bf3a76d23705a84bb880b63)
(cherry picked from commit e5d46399c77d90a5994d190d32ddf445459e1892)
Keep the post-commit child retirement fence without reentering the kernel while the detached spawn transaction is still draining. Source-commit: b7a42cfd2c130d5a46ef6e9c8c16259be7ef4115 Source-commit: 93880b52acbf5619bbab00aaa67107fc83038a14 Source-commit: 108f47fea11ef9e6e4d0833c0db0267d971196d8
601a999 to
63e47d3
Compare
|
Cannot prepare merge: the exact tested PR head needs either a qualified approving review or a fresh |
|
prepare-merge gate failed. Check the workflow run log: https://github.com/Automattic/kandelo/actions/runs/32040652844. Re-apply the ready-to-ship label after fixing. |
|
Cannot prepare merge: latest real non-skipped |
|
prepare-merge gate failed. Check the workflow run log: https://github.com/Automattic/kandelo/actions/runs/32044816031. Re-apply the ready-to-ship label after fixing. |
Phase B-1 matrix build status —
|
| Package | Arch | Status | Sha |
|---|---|---|---|
| icu | wasm32 | failed | — |
| libcurl | wasm32 | failed | — |
| libcxx | wasm32 | failed | — |
| libcxx | wasm64 | failed | — |
| libiconv | wasm32 | failed | — |
| libpng | wasm32 | failed | — |
| libxml2 | wasm32 | failed | — |
| libzip | wasm32 | failed | — |
| openssl | wasm32 | failed | — |
| openssl | wasm64 | failed | — |
| sdl2 | wasm32 | failed | — |
| sdl3 | wasm32 | failed | — |
| sqlite | wasm32 | failed | — |
| sqlite | wasm64 | failed | — |
| zlib | wasm32 | failed | — |
| zlib | wasm64 | failed | — |
| bc | wasm32 | failed | — |
| bzip2 | wasm32 | failed | — |
| coreutils | wasm32 | failed | — |
| cpython | wasm32 | failed | — |
| curl | wasm32 | failed | — |
| dash | wasm32 | failed | — |
| diffutils | wasm32 | failed | — |
| dinit | wasm32 | failed | — |
| erlang | wasm32 | failed | — |
| fbdoom | wasm32 | failed | — |
| file | wasm32 | failed | — |
| findutils | wasm32 | failed | — |
| gawk | wasm32 | failed | — |
| git | wasm32 | failed | — |
| grep | wasm32 | failed | — |
| gzip | wasm32 | failed | — |
| homebrew-bootstrap | wasm32 | failed | — |
| kandelo-sdk | wasm32 | failed | — |
| kernel | wasm32 | failed | — |
| less | wasm32 | failed | — |
| lsof | wasm32 | failed | — |
| m4 | wasm32 | failed | — |
| make | wasm32 | failed | — |
| mariadb | wasm32 | failed | — |
| mariadb | wasm64 | failed | — |
| modeset | wasm32 | failed | — |
| msmtpd | wasm32 | failed | — |
| nano | wasm32 | failed | — |
| ncurses | wasm32 | failed | — |
| netcat | wasm32 | failed | — |
| nginx | wasm32 | failed | — |
| php | wasm32 | failed | — |
| posix-utils-lite | wasm32 | failed | — |
| ruby | wasm32 | failed | — |
| sdl-dsp-test | wasm32 | failed | — |
| sdl2-mixer-playwave | wasm32 | failed | — |
| sed | wasm32 | failed | — |
| spidermonkey | wasm32 | failed | — |
| tar | wasm32 | failed | — |
| tcl | wasm32 | failed | — |
| unzip | wasm32 | failed | — |
| userspace | wasm32 | failed | — |
| vim | wasm32 | failed | — |
| wget | wasm32 | failed | — |
| xz | wasm32 | failed | — |
| zip | wasm32 | failed | — |
| zstd | wasm32 | failed | — |
| bash | wasm32 | failed | — |
| mariadb-test | wasm32 | failed | — |
| mariadb-vfs | wasm32 | failed | — |
| mariadb-vfs | wasm64 | failed | — |
| nethack | wasm32 | failed | — |
| node | wasm32 | failed | — |
| redis-vfs | wasm32 | failed | — |
| spidermonkey-node | wasm32 | failed | — |
| vim-browser-bundle | wasm32 | failed | — |
| nethack-browser-bundle | wasm32 | failed | — |
| rootfs | wasm32 | failed | — |
Auto-generated; replaced on each push. Raw data in the publish-status workflow artifact.
Why
ABI 43 was prepared as a reviewed linear batch, but PR #1264 was
squash-merged. That preserved the final files while discarding the
conceptual commit history, contributor attribution at the commit level,
and the ancestry needed for future bisects and replay-based staging.
This repair restores a reviewable linear history without changing the
already-shipped ABI 43 tree. It also includes the follow-up spawn
liveness fix needed by the msmtpd bottle path.
What changed
POSIX, Homebrew, and continuous integration changes.
commit.
History invariants
the remaining net additions are the replay and repair design/plan
documents.
Validation
Local structural validation completed:
The spawn-liveness change passed its focused 9-test parity suite and the
combined 85-test spawn/exec/deferred/PID suite before history assembly.
At the owner's direction, the assembled branch will use this PR's CI as
its final test gate.
Merge requirement
This PR must be merged with Rebase and merge. Do not squash it.