Skip to content

Release v0.6.0 - #437

Merged
WaylandYang merged 211 commits into
mainfrom
dev
Nov 1, 2026
Merged

WaylandYang merged 211 commits into
mainfrom
dev

Conversation

@WaylandYang

Copy link
Copy Markdown
Contributor

Promotes dev to main for v0.6.0. Merge with a merge commit, not a squash, so main stays an ancestor of dev.

See CHANGELOG.md.

🤖 Generated with Claude Code

jrimmer and others added 30 commits August 16, 2026 22:52
After snapshot restore, the guest's KVM clock reflects the snapshot's
saved time, not the host's current wall time. For snapshots created
minutes or hours ago, the guest clock starts in the past, causing TLS
certificate validation failures, incorrect timestamps, and token
expiry issues inside the guest.

This adds Phase 2.5 to restore_many_with: after the parallel restore
PUT completes, poll each child's guest agent (forkd-agent.py on port
8888) until it responds, then exec `date -s @<host_unix_ts>` to set
the guest clock to the host's current wall time. All children in a
batch are synced in parallel.

This is the cross-platform, cross-kernel-version alternative to the
`clocksource=kvm-clock` boot arg (PR #289), which is x86-only and
forward-only (requires re-baking existing snapshots). This approach
works on any architecture, any Linux kernel, and any existing
snapshot without re-baking.

Best-effort and non-fatal: if the guest agent doesn't come up within
10 seconds, or `date` is unavailable, the sandbox remains usable with
a stale clock. Failures are logged as warnings.

References #288 (stale sandbox VM clock problem).

Signed-off-by: jrimmer <jrimmer@users.noreply.github.com>
Signed-off-by: jrimmer <jason@rimmer.net>
Fix a bug in sync_guest_clocks: the exec args were ["-c", cmd] which
causes subprocess.run to try executing a program named "-c" (fails with
FileNotFoundError). Changed to ["sh", "-c", cmd] so the date command runs
through the shell as intended. The bug was silently masked because clock
sync is best-effort (failures logged as warnings, not propagated) and
the clocksource=kvm-clock boot arg (#289) was correcting the clock on
modern kernels.

Add sync_guest_clocks_corrects_wrong_clock integration test:
- Boots a fresh VM via Vm::boot
- Deliberately sets the guest clock to 2001-09-09 (old_ts=1000000000)
- Calls sync_guest_clocks on the single VM
- Verifies the guest clock is corrected to host wall time (within 5s)
- Verifies the corrected time is NOT the mock old time

This isolates the scripting solution: even if KVM_CLOCK_REALTIME already
corrected the clock, the test forces a wrong clock and verifies the
sync function fixes it. The mock clock underneath is fine — we're
testing the mechanism (date -s via guest agent exec), not the
environment (kernel clock behavior).

The test is #[ignore] (requires Linux + KVM + firecracker + rootfs)
and #[cfg(target_os = "linux")].

Signed-off-by: jrimmer <jason@rimmer.net>
Three findings addressed:

1. Host timestamp captured before polling agent — a child that became
   ready near the 10-second deadline was set ~10 seconds in the past.
   Fix: capture SystemTime::now() per child AFTER ping succeeds,
   immediately before the date -s exec. This minimizes the gap between
   timestamp capture and clock correction.

2. restore_many_with unconditionally performed the agent protocol —
   custom/no-agent snapshots added up to 10 seconds to every restore
   even though failure is only a warning.
   Fix: clock sync is now opt-in via ForkOpts::sync_guest_clocks
   (default false). The daemon's spawn paths (create_sandbox,
   spawn_one_for_workspace) set it to true because all forkd images
   contain forkd-agent. CLI paths keep it false — the CLI doesn't
   know whether the image has an agent.

3. For n>1 with per_child_netns=false, every worker contacted the
   same 10.42.0.2:8888 — the shared IP/MAC topology is fundamentally
   conflicting.
   Fix: restore_many_with now rejects n>1 with per_child_netns=false
   with a clear error message directing callers to use
   per_child_netns=true for multi-child spawns.

Signed-off-by: jrimmer <jason@rimmer.net>
sync_guest_clocks now returns (u128, Vec<ClockSyncOutcome>) instead
of just u128. ClockSyncOutcome is an enum with Synced{child_index}
and Failed{child_index, error} variants, exposed via a new
ForkResult::clock_sync_outcomes field.

Callers (e.g. the controller's create_sandbox) can inspect the
per-child outcomes to decide whether to fail-closed or degrade when
some children have unsynchronized clocks. The controller now logs
clock_sync_failures alongside the existing timing fields.

Also fixes the unused child_index warning from the thread closure
(the index is used in the collection loop, not inside the thread).

Tests:
- clock_sync_outcome_construction_and_matching: verifies enum
  discriminants, field access, and the filter-for-failures pattern
  callers use to decide fail-closed behavior.
- sync_guest_clocks_corrects_wrong_clock (#[ignore] integration):
  now also asserts the outcome vec contains exactly one Synced
  entry for the single child.
- fork_result_has_clock_sync_ms_field: updated to include the new
  clock_sync_outcomes field.

Signed-off-by: jrimmer <jason@rimmer.net>
Two CI/coverage gaps from the review:

1. Clippy (rust CI was red): the sync_guest_clocks child loop used
   .enumerate() and discarded the index (clippy::unused_enumerate_index).
   Drop the .enumerate() and iterate children directly.

2. Behavioral coverage for the three review asks:
   - multi-child address/mode: restore_many_with_rejects_multi_child_shared_ip
     unit test proves n>1 with per_child_netns=false is rejected before any
     Firecracker spawn or filesystem side effect.
   - no-agent/disabled-sync latency: fork_opts_sync_guest_clocks_defaults_disabled
     unit test proves the opt-in default keeps sync off (no added latency for
     custom/no-agent images).
   - delayed agent readiness (timestamp freshness):
     sync_guest_clocks_delayed_agent_readiness (#[ignore] integration) boots a
     VM, calls sync_guest_clocks immediately (no pre-wait) so it must poll for
     the not-yet-ready agent, then asserts the guest clock matches host wall
     time within 3s — a stale pre-poll timestamp would drift by the poll
     duration.

Signed-off-by: jrimmer <jason@rimmer.net>
WaylandYang's final review round (round 5, 2026-08-14) flagged the
delayed-agent-readiness integration test as non-deterministic: it
called sync_guest_clocks immediately and only checked the final drift
plus an upper elapsed bound, so it never guaranteed that agent
readiness was actually delayed beyond the accepted 3-second drift. If
the agent came up quickly, the old pre-poll-timestamp implementation
still passed the test.

Extract the timestamp-placement logic from sync_guest_clocks into
sync_one_child_clock(ping, now, set_clock), which accepts injectable
readiness-probe / timestamp-source / clock-setter closures. The
production call site wires these to ping_at/ping_in_netns,
SystemTime::now, and the date -s exec; the test wires them to a
deterministic probe that fails for 4 seconds (> the 3s accepted drift)
then succeeds.

The new unit test sync_one_child_clock_timestamp_fresh_after_delayed_ping:
  1. asserts the induced poll delay (4s) exceeds the accepted drift (3s),
     so the freshness assertion is meaningful,
  2. asserts the timestamp passed to set_clock was captured AFTER ping
     succeeded (now_called_at >= probe_ready_at, freshness_gap < 3s) --
     a pre-poll-timestamp implementation would fail by ~4s,
  3. asserts set_clock was actually invoked with that fresh timestamp.

No VM, guest agent, or KVM is required -- it runs in the unit-test suite
on every platform. The end-to-end mechanism (real agent + date -s) is
still covered by the #[ignore] integration test
sync_guest_clocks_corrects_wrong_clock.

Verified on Linux (rust 1.83, x86_64): cargo clippy clean for
forkd-vmm/controller/cli/uffd; cargo test -p forkd-vmm --lib passes
48 passed, 0 failed, 3 ignored.

Signed-off-by: jrimmer <jason@rimmer.net>
Apply the three cross-reviewer findings from the ce-code-review pass
(5 reviewers: correctness, testing, maintainability, reliability,
adversarial):

1. Timeout error dropped the agent address (P2, safe_auto) — flagged by
   correctness, testing, and adversarial. In multi-child restores the
   operator could not tell which child's agent failed to come up. Now
   wrapped at the call site: `.map_err(|e| e.context(format!(
   "guest agent at {addr} did not come up")))`.

2. No unit test for the timeout path (P2, manual) — flagged by testing,
   reliability, and adversarial. Only the #[ignore] integration tests
   exercised it, and the 10s production timeout made a dedicated unit
   test impractical. The timeout is now an injected parameter on
   sync_one_child_clock, and sync_one_child_clock_times_out_when_probe_
   never_succeeds uses a 500ms injected timeout to verify the bail
   message and that now()/set_clock are never called on the timeout
   path.

3. No unit test for set_clock failure propagation (P2, manual) —
   flagged by testing, reliability, and adversarial.
   sync_one_child_clock_propagates_set_clock_failure verifies a
   set_clock error propagates verbatim and that now() was still called
   (timestamp captured before the failed set).

Acknowledged (P3 advisory, no action): the 4s induced delay makes the
freshness test the slowest non-ignored unit test; wall-clock second
granularity in the freshness assertions is robust given the 4s>3s
headroom and the compile-time invariant guard.

Verified on Linux (rust 1.83, x86_64): cargo clippy clean for
forkd-vmm/controller/cli/uffd; cargo test -p forkd-vmm --lib passes
50 passed, 0 failed, 3 ignored.

Signed-off-by: jrimmer <jason@rimmer.net>
Signed-off-by: jrimmer <jason@rimmer.net>
fix(vmm): best-effort guest clock sync after snapshot restore (Option B)
reconcile() took K+4 separate registry-mutex acquisitions for a pass
that prunes K stale sandboxes: one lock per removed id, plus one each
for the stale scan, the live-id snapshot, and the workspace walk. Two
handler calls interleaved between the scan and the per-id removal
could make the reported prune count diverge from what was actually
removed.

Collect the stale ids, remove them, and walk the workspaces for the
Running -> Stale transition all under one `self.inner.lock()`
acquisition. `pid_alive` is a `/proc` stat and takes no lock we care
about, so holding the registry mutex across it is fine. flush()
behavior is unchanged: it still runs once, after the lock is dropped,
only when something changed.

Adds coverage for reconcile()'s prune-count and workspace-transition
behavior (previously untested): a dead sandbox is pruned and its
Running workspace flips to Stale with live_sandbox_id cleared; a
sandbox with no recorded pid is left alone and its workspace stays
Running; a Suspended workspace is never touched; and a no-op pass
does not rewrite state.json.

Fixes #310

Signed-off-by: Amir Fathi <amirfathi.me@gmail.com>
…ock-310

fix(controller): run Registry::reconcile under a single lock
Squash of 10 commits on fix/restore-path-resilience for rebasing onto
current dev (which now contains #281's shared-TAP lifecycle).

Original commits:
- 858f63b fix(vmm): restore-path resilience for issue #301
- ce3d800 fix(vmm): apply review fixes for #301
- 0d73362 fix(vmm): make DEFAULT_SOCKET_WAIT_SECS pub for cross-crate use
- e820848 fix(controller): use Vm::pid() accessor, not private field
- 1ddf40c fix(controller): drop needless borrow in workspace orphan scan call
- ca52820 fix(vmm): partial-success contract + concurrent socket wait for restore_many_with
- 9e863de fix(controller): expose orphan-firecracker detection as a metric
- 2aca840 test(vmm): integration test for partial-success restore
- 9342e7a test(vmm): accept Restore or SocketWait phase in partial-success test
- f2751b9 fix(vmm): review fixes for #301 — indexed success, valid test topology, docs

Signed-off-by: jrimmer <jason@rimmer.net>
Signed-off-by: jrimmer <jason@rimmer.net>
Review #302: restore_many_with flattening children via
children::flatten() renumbered ClockSyncOutcome.child_index, so a
survivor past a failed hole (spawn/socket-wait) was reported as
0..n instead of its original 1-based within-batch slot, breaking
the resource correlation that ForkChild/RestoreFailure promise.

- sync_guest_clocks now takes &[(usize, &Vm)] where the usize is the
  original 1-based within-batch slot; outcomes echo that slot in both
  ClockSyncOutcome::Synced/Failed and the tracing child spans.
- Add live_children_with_indices(): the single index-preserving
  mapping (enumerate + filter_map, i+1) used by restore_many_with,
  extracted so the invariant is unit-testable without KVM.
- Regression: live_children_with_indices_preserves_survivor_slot_  across_hole proves [None,Some,None,Some] -> slots [2,4], not [0,1].
- Update the single-child clock-sync integration test to pass slot 1
  and assert child_index == 1.

Signed-off-by: jrimmer <jason@rimmer.net>
Map a line directly following the #Cons list was parsed as an
unindented list continuation (-D warnings / clippy::doc-lazy-continuation).
Separate it with a blank doc line.

Signed-off-by: jrimmer <jason@rimmer.net>
fix(vmm): restore-path resilience — configurable timeout, per-VM partial-failure reporting, pre-restore orphan detection
Issue #298: when the controller crashes or restarts while Firecracker
processes are still alive, the restarted controller cannot reconstruct
resource ownership for orphaned VMs. Both the NetnsAllocator (active
set) and shared_tap_owner are initialized empty, so a new spawn can
reserve an orphaned VM's netns index or claim the shared tap even
though the orphan still owns it.

Fix: add `Registry::kill_orphans()` called after `reconcile()` on
startup. After reconcile() prunes dead-PID entries, any remaining
sandbox entries have alive PIDs but no live_vms handle — they are
unmanageable orphans. kill_orphans() kills each orphan (after
verifying the PID belongs to a Firecracker process via /proc/<pid>/comm
to guard against PID reuse), prunes the registry entry, and marks
workspaces whose sandbox was killed as Stale.

After this, the NetnsAllocator (empty active set) and shared_tap_owner
(None) are safe: no orphaned VM holds a netns index or the shared tap.

Closes #298

Signed-off-by: jrimmer <jason@rimmer.net>
Address all P0 and P1 findings from the 7-reviewer ce-* panel:

P0: kill_pid failure no longer prunes the registry entry. ESRCH (process
already dead from TOCTOU race) is distinguished from real errors (EPERM
etc.). On ESRCH, the entry is safely pruned. On real errors, the entry
is kept in the registry to prevent the exact resource collision issue

P1 fixes:
- Wait for process death after SIGKILL (5s bounded timeout polling
  /proc/<pid> disappearance). A D-state process stuck on I/O can hold
  netns/tap resources past the kill return; if it doesn't exit within
  the timeout, the registry entry is kept.
- pid_is_firecracker uses exact match (== "firecracker") instead of
  substring match to avoid false-positive kills of unrelated processes.
- Separate KillOrphansResult struct (killed, pruned_stale, kill_failed)
  replaces the misleading single usize counter. The startup log now
  reports all three counts.
- Extract mark_stale_workspaces() helper to eliminate the ~22-line
  duplicated workspace stale-marking block between reconcile() and
  kill_orphans().
- kill_orphans is pub(crate) with a doc invariant: "MUST only be called
  before any Vm handle is held" — prevents future post-startup misuse.
- Defensive pid <= 1 guard in pid_is_firecracker (defense-in-depth
  against corrupted state.json).
- ESRCH downgraded from error! to debug! (benign TOCTOU race).
- SAFETY comment added on unsafe libc::kill.

New tests: pid:None entries skipped, multiple workspaces same sandbox,
persistence to disk after kill_orphans, reconcile+kill_orphans integration.

Signed-off-by: jrimmer <jason@rimmer.net>
WaylandYang round 4 found that kill_orphans correctly keeps registry
entries on EPERM/timeout (kill_failed > 0), but run_daemon only logs
the count and continues. The NetnsAllocator active set and
shared_tap_owner start empty, so the next spawn can reuse the failed
orphan's netns index or tap lease — the exact collision #298 prevents.

Fix: run_daemon now aborts startup when kill_failed > 0, forcing
operator intervention rather than silently admitting conflicting
spawns. The error message names the count and tells the operator to
kill the orphans manually and restart.

Also updated the pid_is_firecracker doc comment to be honest about the
PID-reuse limitation: the comm check is a defense-in-depth heuristic,
not complete PID-reuse protection. A robust fix would use pidfd
signaling plus durable identity validation (process start time recorded
with the registry entry at spawn time, compared on startup). That is
tracked as a follow-up.

Added test: kill_orphans_result_kill_failed_contract verifies the
KillOrphansResult contract (kill_failed == 0 on empty registry, so
baseline startup does NOT abort).

Signed-off-by: jrimmer <jason@rimmer.net>
…decision

The startup abort logic was inline in run_daemon, making it
impossible to unit-test the kill_failed > 0 → bail decision path.
Extracted into pub(crate) check_orphan_kill_result(&KillOrphansResult)
-> Result<()> so the abort contract is directly testable.

Tests:
- check_orphan_kill_result_aborts_on_kill_failure: verifies that
  kill_failed=1 returns Err with "aborting startup" message, and
  kill_failed=0 returns Ok even with killed/pruned entries.
- check_orphan_kill_result_error_contains_count: verifies the error
  message contains the exact kill_failed count so operators know
  how many orphans need manual intervention.

The existing kill_orphans_result_kill_failed_contract test
(zero-baseline) is retained as documentation of the baseline
contract. The new tests exercise the actual abort decision logic.

Signed-off-by: jrimmer <jason@rimmer.net>
The dead-last_err cleanup dropped the declaration and the unreachable
expect, but the `last_err = Some(e)` assignment (guarded by the
tap/cgroup-busy warn block on main) survived the rebase and referenced a
now-missing binding. Remove it — the loop already returns on the final
attempt, so the assignment is dead.

Signed-off-by: jrimmer <jason@rimmer.net>
Replace the comm=="firecracker" check with durable process identity
verification and pidfd signaling, addressing review r6 (WaylandYang)
and the ce-code-review security findings.

The comm check was spoofable (prctl PR_SET_NAME) and could not
distinguish two Firecracker processes sharing a PID over time — a
same-name PID-reuse attack would kill a legitimate new Firecracker,
and on a multi-tenant host a cross-tenant DoS was possible. The
TOCTOU window between the check and kill(2) was also acknowledged.

Changes:

1. SandboxInfo::proc_starttime (serde default) — persists the process
   start time (field 22 of /proc/<pid>/stat, clock ticks since boot)
   captured at VM registration. Backward-compatible: old state.json
   files without this field deserialize to None.

2. read_proc_starttime — robust parse of /proc/<pid>/stat that handles
   the parenthesized comm field (field 2, can contain spaces/parens) by
   splitting at the last ')' and indexing field 22 in the remainder.

3. process_identity_matches — compares the live start time against the
   recorded one. Returns Match / PidReuse / Dead / Unknown. PidReuse
   means the original process exited and the PID was recycled — prune
   the stale entry without killing. Unknown (no recorded start time,
   /proc unreadable, or off-Linux) fails closed.

4. pidfd_open + pidfd_send_kill — signal through a pidfd opened AFTER
   the identity check, closing the TOCTOU window: the fd is pinned to
   the verified process, so a PID reuse between check and signal cannot
   redirect the kill. Uses raw libc::syscall with SYS_pidfd_open /
   SYS_pidfd_send_signal (Linux 5.3+; libc 0.2.x exposes the syscall
   numbers but not named fns).

5. comm_is_firecracker — retained as a secondary confirmation after
   the primary start-time check passes, to catch corrupted state.json.
   Not a security boundary.

6. kill_orphans reworked: Match → pidfd kill + wait_for_death; PidReuse
   → prune stale (no kill); Dead → prune stale; Unknown → fail closed
   (keep entry, increment kill_failed so run_daemon aborts startup).

Tests updated to reflect the new fail-closed semantics:
- kill_orphans_prunes_alive_pid_entries now uses a mismatched start
  time (u64::MAX) to trigger PidReuse → prune, not the old None path.
- kill_orphans_fails_closed_on_unknown_identity (new): alive PID with
  proc_starttime: None → fail closed, entry kept, kill_failed=1.
- kill_orphans_prunes_same_name_pid_reuse_without_killing (new
  regression): the same-name Firecracker PID-reuse attack the reviewer
  flagged — a mismatched start time must prune without killing, even
  if comm were "firecracker".
- All other kill_orphans tests updated to use mismatched start times
  where they previously relied on the comm-check prune path.

Verified on Linux (rust 1.83, x86_64) via Docker: clippy clean for
forkd-controller; cargo test --all-features passes 104 unit + 8
integration tests, 0 failed.

Signed-off-by: jrimmer <jason@rimmer.net>
Address ce-code-review findings (security P1, testing P1, adversarial
P2/P3, plus the cross-reviewer coverage gaps) on the durable-identity
rework from 2e275fe.

1. Residual TOCTOU window (security P1, adversarial ADV-1): pidfd
   closes only the open→signal window, NOT the check→open window. The
   original could exit and the PID could be reused by another
   Firecracker between process_identity_matches (which reads
   /proc/<pid>/stat) and pidfd_open. Fix: after pidfd_open succeeds,
   re-read the live start time and compare to the recorded one. A
   mismatch means the PID was reused between the check and the open →
   prune without killing (PidReuse). The pidfd pins the process at
   open time, so /proc/<pid> reflects the pinned process. This closes
   the check→open window; the comm check is now a pure sanity check,
   not a load-bearing defense. Design comment corrected to not
   overstate the guarantee.

2. Off-Linux comment/behavior mismatch (adversarial ADV-2): the
   module comment claimed off-Linux "prunes without killing" but
   Unknown actually fails closed (keeps entry, increments kill_failed,
   run_daemon aborts). Fixed the comment to state the real behavior.

3. Cross-platform test guards (adversarial ADV-3): the PidReuse-
   asserting tests (mismatched starttime → pruned_stale) now carry
   #[cfg(target_os = "linux")] because off-Linux returns Unknown
   (kill_failed), not PidReuse. The cross-platform fail-closed test
   (proc_starttime: None → Unknown) stays ungated.

4. Match-arm coverage (testing P1, +reliability/security/adversarial):
   new kill_orphans_match_arm_kills_verified_child_via_pidfd spawns a
   real child, records its true start time (Match), and asserts the
   secondary comm_is_firecracker check fails closed for a
   non-firecracker process — proving the Match arm is entered and the
   comm gate works as defense-in-depth. (Killing a real Firecracker
   would require a Firecracker binary; the comm gate is the
   testable boundary here.)

5. start-time parser unit test (testing P2): read_proc_starttime now
   has a direct test (parses our own /proc/<pid>/stat, rejects pid<=1,
   returns None for nonexistent PID).

6. Backward-compat serde test (testing safe_auto): new
   proc_starttime_defaults_to_none_when_absent_in_old_state_json
   deserializes a hand-written pre-r6 state.json (no proc_starttime
   key) and asserts #[serde(default)] yields None — the contract that
   lets existing deployments upgrade without wiping state.

Verified on Linux (rust 1.83, x86_64) via Docker: clippy clean for
forkd-controller; cargo test --all-features passes 107 unit + 8
integration tests, 0 failed (3 new tests pass).

Signed-off-by: jrimmer <jason@rimmer.net>
…ding comments

Review r8 (2026-08-20) found that
kill_orphans_match_arm_kills_verified_child_via_pidfd did not do what
its name claimed: it spawned 'sleep' (comm='sleep'), so the secondary
comm_is_firecracker check failed closed and the irreversible success
path (pidfd_send_kill -> wait_for_death -> registry removal) was never
executed.

- Rename the existing test to
  kill_orphans_match_arm_comm_mismatch_fails_closed so its name and
  doc accurately describe the comm-gate fail-closed branch it covers.
- Add kill_orphans_match_arm_kills_verified_firecracker_via_pidfd,
  which spawns a disposable process whose /proc/<pid>/comm is literally
  'firecracker' (a copy of 'sleep' renamed to 'firecracker'), records
  its true start time, and asserts killed == 1, kill_failed == 0,
  pruned_stale == 0, and the registry entry is removed — exercising the
  full irreversible pidfd kill path.

Also correct two misleading comments flagged in the same review:
- SandboxInfo::proc_starttime claimed legacy None state may use
  comm-only verification; the implementation fails closed (keeps the
  entry, increments kill_failed) and never does comm-only kill.
- IdentityCheck::Unknown claimed the caller chooses between a
  comm-only kill or prune; kill_orphans keeps the entry and aborts
  startup via kill_failed.

Signed-off-by: jrimmer <jason@rimmer.net>
Signed-off-by: jrimmer <jason@rimmer.net>
Addresses review #299 (both blockers) and rebases onto dev with #312's
single-lock reconcile preserved.

Blocker 1 — cross-reboot false Match (PID+starttick collision):
  proc_starttime is ticks since boot, only unique within one host boot.
  The registry persists across reboots, so after a reboot an unrelated
  Firecracker can share both the numeric PID and the boot-relative start
  tick of a recorded entry, and a bare starttime check would SIGKILL the
  wrong process. Persist /proc/sys/kernel/random/boot_id at registration
  (SandboxInfo.boot_id) and gate the identity check on it:
    - different boot id -> PidReuse (prune WITHOUT signaling; the old
      process cannot still exist across a reboot)
    - missing recorded boot id (legacy state.json) -> Unknown (fail closed)
    - live boot id unreadable -> Unknown (fail closed)
  New read_boot_id() helper; recorded at all three production VM
  registration sites in http.rs.

Blocker 2 — pid: None entries recreated the #298 collision risk:
  Such rows were skipped by both reconcile and kill_orphans, so startup
  succeeded with an empty allocator/shared-tap ownership while a live VM
  may still hold those resources. kill_orphans now counts retained rows
  without a PID into KillOrphansResult.unresolved, and
  check_orphan_kill_result aborts startup on unresolved > 0 (fail closed).

Rebase: preserved #312's single-lock reconcile (whole in-memory pass under
one registry lock, including inline workspace-stale marking); kept
mark_stale_workspaces only for kill_orphans' use after pruning.

Tests:
  - kill_orphans_does_not_kill_across_boot_id_mismatch: matching PID +
    real start time but a DIFFERENT recorded boot id -> pruned without
    signaling (killed==0, pruned_stale==1).
  - kill_orphans_counts_pid_none_entries_as_unresolved (replaces
    kill_orphans_skips_pid_none_entries): pid:None rows surface as
    unresolved and block startup via check_orphan_kill_result.
  - check_orphan_kill_result_aborts_on_unresolved: startup-decision
    regression.
  - Existing kill-path tests set boot_id to the current boot id where
    they exercise the Match/PidReuse path; the legacy fail-closed test
    keeps boot_id: None.

Signed-off-by: jrimmer <jason@rimmer.net>
The rebase onto post-#302 dev merged both sides of the busy-retry block
textually but not semantically: in the Err(e) arm, last_err = Some(e)
(and the final_attempt variant Some(e.context(...))) both move e, then the
trailing tracing::warn!(error = %e, ...) borrowed the moved value. #302 had
moved that warn into the else branch ahead of the assignment.

The arm already computes let msg = format!("{e:#}") for busy-classification,
so use error = %msg in the warn to avoid re-borrowing e after the moves.
…phan-reaping

kill_orphans_match_arm_kills_verified_firecracker_via_pidfd spawned its
firecracker-named 'sleep' via sh -c '<bin> 30 & echo $!' and relied on the
backgrounded process being reparented to init, so wait_for_death would see
/proc/<pid> disappear right after SIGKILL. On the GitHub Actions runner that
detached child is reaped almost immediately (diagnostic showed state=Z ppid=1
at t=0ms, dead by 5ms), so process_identity_matches returned Dead,
kill_orphans pruned it, and the test failed with pruned_stale=1 instead of
killed=1.

Spawn the process as a DIRECT child the test owns (the runner cannot reap an
owned, non-orphaned child), and reap it via a small reaper thread that calls
Child::wait(). wait_for_death (polling /proc/<pid>) then observes the SIGKILL
reaping and the Match arm reports killed=1 as intended.
…p (review #299 r9)

Legacy state.json rows (no recorded boot identity via serde defaults)
took the generic Unknown path on first restart after upgrade and aborted
startup with an EPERM-style message describing neither cause nor fix.

- IdentityCheck::LegacyUnverifiable: missing recorded starttime OR boot
  id is reported distinctly from a genuinely unverifiable environment.
- KillOrphansResult.legacy_unverified counts legacy rows separately
  from kill_failed; run_daemon logs include the new counter.
- check_orphan_kill_result emits a dedicated upgrade diagnostic (kill
  the recorded PID manually or remove the entry once verified dead),
  distinct from the kill-failure and pid-less-row messages.
- Off-Linux, identity can never be verified, so the gate skipped any
  registry row and aborted dev-box startup; the abort decision is now
  extracted into a pure cross-platform orphan_abort_reason(), the Linux
  gate fails closed on it, and the off-Linux gate logs the counts and
  allows startup (review #299 non-blocking).
- Corrected the wrong pidfd comment: /proc/<pid> reflects whoever
  currently holds the PID; the post-open starttime re-read is what
  makes the reuse check safe.
- kill_orphans folds its two entry lock acquisitions into one guard,
  matching the #312 single-lock invariant.

Tests: legacy rows count as legacy_unverified (not kill_failed); abort
distinctness (legacy vs kill-failure diagnostics); Linux-gated gate
wiring test; existing abort-contract tests retargeted to the pure
decision function so they run on every platform.
Floating-Y and others added 29 commits October 31, 2026 02:04
* docs: correct security policy for current runtime

* docs: clarify sandbox security limitations

* docs: retain gvisor isolation warning

* docs: tighten the security policy

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: WaylandYang <145302500+WaylandYang@users.noreply.github.com>

---------

Signed-off-by: WaylandYang <145302500+WaylandYang@users.noreply.github.com>
Co-authored-by: WaylandYang <145302500+WaylandYang@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…#408)

The `sandstorm` CLI and `sandstorm-controller` left the tree, but the
recipes, Hub, Arch package, Kubernetes manifest, host scripts and docs
that needed them stayed, and none can run. The MCP server's
`create_snapshot` tool called a route that no longer exists.

docs/API.md described that daemon: the wrong port and flags, routes that
are gone, and none of the node, template or tier routes. It now follows
the router.

Signed-off-by: WaylandYang <145302500+WaylandYang@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* feat(sdk-python)!: drive Sandbox through the master's API

Sandbox started sandboxes by running the `sandstorm` CLI, which left the
tree in v0.6, so the package's default export failed on first use.

It now creates through Controller, from a template or a tag, waits for a
node to report the sandbox running, and sends the creation token on exec,
eval and ping. Sandbox.connect wraps a sandbox that exists. eval returns
what the snippet printed, since the master runs it through the template's
eval_argv rather than a warmed interpreter.

Controller gains create_sandbox, and exec and eval now wait for as long as
the command may run rather than Controller.timeout.

test_send.py and smoke-sdk.py went with the TCP path and result_json they
tested.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(api): eval returns what the snippet printed

The example showed a bare expression returning its value, but the master
runs the snippet through eval_argv, so 2+2 prints nothing.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
When `runsc start` failed, for example on a template whose program is not
in its rootfs, the container stayed stopped with its sandbox process alive
in the cgroup. `observe` reported that as a running sandbox, so the master
recorded an address, cleared the failure and never retried, and commands
sent to it failed or timed out.

Runsc now removes the container when create, start or restore fails, as
Firecracker already stops its VM. The sandbox is then not observed, so it
is retried and quarantined with `last_error` set. A failed `runsc create`
on a leftover container also clears the way for the next attempt.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
The node removes a container that fails to start, so the sandbox is
quarantined and Sandbox raises instead of returning. A program that starts
and then exits is the case left.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
The tiers table stretched to full width on one long State cell, so its
headers wrapped. Shorter cells let it fit, with memory and latency aligned.

docs/OPTIMIZATIONS.md lists what was optimized for gVisor, for Firecracker
and for both, with the before and after from when each change was made.
Key features links to it.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…ck (#414)

A sandbox its node put to disk to make room still had its REJECT in the
host's OUTPUT when it came back, and that rule only goes once set_tier has
finished. sync_clock reaches the agent through OUTPUT, so it spent its whole
2 s on refused connects and gave up, with the wake waiting behind it.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* bench: latency under gVisor, KVM and PVM on one build

workloads.py asked runsc whether a sandbox had reached the disk, so under
Firecracker it returned at once and measured hot sandboxes. It reads the
sandbox's cgroup now, which works under either backend. The frozen wake in
the lifecycle table was ~10 ms; measured through the gateway it is ~0.1 s.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* bench: PVM again, with counts that fit its host

The first round asked 2.4 GB for 14 and 40 Firecracker guests, so everything
waited on swap, and the column read as PVM being slow. With 8 and 20 it is not.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* bench: PVM's kept in memory, and what its 2 s wakes were

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
….0 (#420)

Controller() went to http://127.0.0.1:8889, the v0.5 daemon's address. A
master listens on https://127.0.0.1:8080. __version__ said 0.5.1 where the
package says 0.5.3, and a test now keeps the two together. The package's
description and keywords still called it a microVM primitive.

Also drops stale lines: the ADR index said Firecracker had not run on KVM,
and the sdk-python CI comment named tests that are gone.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* feat(node): start a batch's creates side by side

A node started the sandboxes of a batch one after another: 45 ms each on
Firecracker and 96 ms on gVisor, whatever the machine. A run of creates
now goes one per CPU at once, each under its own sandbox's lock, and a
create promises its memory as it is let through, so a burst is not judged
against one reading of free memory.

The namespace pool follows memory, 32 to 256: past it a burst waited on
the RTNL lock, and 100 children took as long side by side as one after
another.

Running 100 at a time found two bugs that serial runs have too. A repeat
of a finished destroy took down or deleted a veth by a name that was by
then the pool's or another sandbox's, and a sandbox whose network never
arrived stayed running where nothing could reach it, unretried.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* bench: why the gVisor create rows were slow

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* fix(node): a warm image does not outlive the host files under it

On gVisor a sandbox runs on the host's /usr, bound in, and a warm image
holds a program laid out for the files it had mapped. unattended-upgrades
replaced python3.12 and libc under an 18-hour-old image, and every sandbox
restored from it segfaulted 200 ms later. The node reported it gone, the
master asked again, and the restore fell back to a cold boot: every create
ran twice, and nothing said so. It read as warm creates being slow in a
burst, 160 ms each against 37 ms for a fork.

The key now follows when the bound directories last changed, one level
down, so an upgrade seeds a new image. That cannot see everything, so a
sandbox created again within 30 s of a warm start also drops the image.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* bench: a warm image under an upgraded host, before and after

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs: a line between two changelog entries

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
A snapshot's image was deleted as soon as runsc restore returned, before
the program had run. With a library replaced under a snapshot, as
unattended-upgrades does, the program then died, the master recreated the
sandbox from its template, and the caller got a 200 from a sandbox with
none of its memory. No error anywhere.

The image now stays until the sandbox has lasted a minute. One that dies
sooner reads as a snapshot again, fails with a reason, and gets a real try
at most once a minute, so it comes back with its memory if the files do.
A fork of a tag that dies the same way stops with that error rather than
restarting for ever. The marker is on disk, so a node restart keeps it.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* fix(node): a sandbox waiting to retry costs no polls

The master answers a poll at once if it has a command, and diff issues one
for everything that has not converged, including what the node is backing
off on. A node that dialled in skipped it and polled again straight away
until the retry was due: about 350 polls a second and 40% of a core for
one failing sandbox, with a master log line each time. The driver for a
listening node already guards against this with same_batch.

A node now says what it is putting off and for how long. The master
leaves those out and parks the poll, and hands them over once they are
due. Commands for other sandboxes still arrive at once. Forgetting the
failures of a sandbox the master stopped asking about skips these, or
their attempts would never count up to quarantine.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs: drop a comment about a return value that has gone

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
#425)

a_restore_that_fails_removes_the_container failed on dev and on a PR, with
calls "delete " where "restore delete " was expected: runsc never ran. The
test wrote the script from this process, and a test forking on another
thread keeps the write descriptor until it execs, so exec of the script
failed with ETXTBSY. In the same shape on Linux, 176 of 32,000 execs failed.
With a child writing the script, none did.

The real runsc is not written by the process that runs it, so this is the
test's own race.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
A closed listener lives on in any process another thread has just forked, until
it reaches exec. Rebinding its port then fails with EADDRINUSE, and the sandbox
is unreachable until the next report. The gateway tries again for up to 100 ms.

Found through tests that failed about one full run in ten, which also turned the
rust check on dev red. Two others had their own causes:

- backend.rs ran a fake runsc it had just written, and a child forked by another
  test held the write descriptor: ETXTBSY. A child process installs it now.
- a_fork_in_a_group_can_start_its_fetch fetched from an address refused at once,
  so the fetch could fail before the second fork asked. It now fetches from a
  listener that never answers.

Signed-off-by: WaylandYang <145302500+WaylandYang@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
A sandbox could reach nothing beyond its node and nothing could change that.
Each now has a policy: internet for public addresses, allow and deny for IPv4
addresses and CIDRs. The console sets it per sandbox and sets the default for
new ones, and the API and the E2B front door take it.

The default stays closed, so nothing changes until someone opens it. A sandbox
with no exception costs no rule. A node installs exceptions per sandbox in one
iptables-restore transaction and reports what it installed, so a restarted one
gets them back by the usual diff.

Outbound traffic with no exception is now refused rather than dropped, so a
closed sandbox looking a name up fails in about 0.1 s, not 5 s. Under gVisor a
rootfs with no usable resolver gets public ones in the sandbox's layer, as
Firecracker guests already did.

Signed-off-by: WaylandYang <145302500+WaylandYang@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
A template was files an operator put on every node by hand. Now a node pulls
an image from its registry itself, with no Docker, and builds what its backend
reads. The master asks every node, including ones that join later, and what the
image says to run becomes the template. The console has a form for it and a
table of how each node got on.

A layer is not unpacked by tar: it guards an archive against itself, not against
the layer before it, and one layer's link would let the next write anywhere on
the host. Each path is resolved as a process chrooted into the rootfs would
resolve it. Blobs must hash to their digest.

Signed-off-by: WaylandYang <145302500+WaylandYang@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: WaylandYang <145302500+WaylandYang@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…rify it by name (#421)

* fix(sdk)!: reach a master at its default address from TypeScript and MCP

Both still defaulted to http://127.0.0.1:8889, the v0.5 daemon's address, and
neither could trust the private CA a master signs its certificate with. They
now default to https://127.0.0.1:8080 and read SANDSTORM_CA, as the Python
Controller does, with no way to turn verification off.

Node's global fetch has no CA option short of an undici dependency, so a CA
goes through a small fetch over node:https. It states rejectUnauthorized,
which otherwise follows NODE_TLS_REJECT_UNAUTHORIZED, and refuses a CA file
with no certificate, since an empty `ca` means Node's public roots.

CI ran only the MCP stdio test, so the new TLS tests would never have run.
It now runs the whole directory.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(sdk)!: bring the TypeScript SDK and MCP server up to the v0.6 API

Neither could start anything on a new master: both needed a snapshot tag,
and a tag only comes from branching a sandbox. Both now start from a
template, as the Python SDK does, and wait until a node reports the sandbox
running, removing it if it never does. An exec sent earlier fails.

What the master refuses or ignores is gone rather than advertised: diff,
measure_diff, mode, wait, live_fork, hugepages and prewarm. Branch needs a
tag. MCP no longer sends memory_limit_mib=256, which overrode every
template's memory. eval returns what was printed. The TypeScript types
describe the master's responses.

Exec, eval and branch now wait as long as the master may take, where a flat
60 s timed out first. touch and list_templates are new in both, since a
branch needs a hot sandbox.

CodeQL flagged two test lines: the MCP stand-in allowed TLS 1.0, and the
NODE_TLS_REJECT_UNAUTHORIZED test assigned the variable. The first now
requires TLS 1.2. The second is what that test exists to check, and uses
vi.stubEnv, which also restores it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix!: let clients verify a master by name, and close the remaining v0.6 gaps

A master's certificate named only 127.0.0.1, localhost and the bind address,
so a client on another machine that dialled a hostname was refused, and with
verification not optional it had no way in. --tls-san adds names. 0.0.0.0 is
no longer written into the certificate, since nobody can dial it, and a master
bound to it with no --tls-san warns. Nodes pin the CA and are unaffected.

A branch ignored mode="diff", so a caller got a full branch and was not told.
Everything but "full" now answers 410, as docs/API.md said.

The Python Controller drops what v0.6 refuses, as the other two clients did,
needs a tag, and waits as long as a branch may take. docs/API.md gains
GET /ca.pem. sandstormd.service and the Dockerfile move to port 8080, and the
service keeps its token in pki/api-token like any other master.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix: keep the bind address on the certificate, and fix what review found

The last commit took 0.0.0.0 off the master's certificate as a name nobody
can dial. A node in the same process dials exactly that, and only its
registration is pinned: every later call verifies the name. So
`--master --node --bind 0.0.0.0` could not poll, and fleet.sh's curl to the
bind address failed the same way. It stays. The help text and CHANGELOG said
nodes do not need --tls-san, which was wrong for the same reason.

--tls-san values are trimmed and blanks dropped. MCP spawn_sandboxes removes
its sandboxes on any failure, since their tokens exist nowhere else, and its
timeout covers the whole batch. delete_snapshot encodes the tag. A failed read
of the CA file is not cached. eval errors carry the exit code, since the error
text can be empty. Sandbox.create no longer lists every CreateOptions field.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* docs(contributing): name every branch feature/<short-name>

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: WaylandYang <145302500+WaylandYang@users.noreply.github.com>

* docs(contributing): apply the branch rule to features only

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: WaylandYang <145302500+WaylandYang@users.noreply.github.com>

---------

Signed-off-by: WaylandYang <145302500+WaylandYang@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Rust takes about 5.5 minutes in CodeQL, nearly all of it extraction and
queries over the whole crate, with no cache to use. 25 of the last 60 PRs
touched nothing under crates/ or Cargo.*, and still waited for it.

Default setup cannot filter by path, so this moves to workflows of our own.
Rust runs on PRs that touch it, the other languages skip docs-only PRs, and
a new push to a PR cancels the scan of the last one. Pushes to dev and main
and the weekly run still scan everything. The categories match default
setup's so alerts carry over.

Default setup has to be turned off before these can upload.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
a_wake_waits_for_a_tier_change_in_progress starts eight wakes, but how many may
wait depends on the machine's CPUs, so it failed on two. It now sets the limit
it needs.

Signed-off-by: WaylandYang <145302500+WaylandYang@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: WaylandYang <145302500+WaylandYang@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
The host reaches the agent at the address the guest was booted with, so it binds
that, read from the kernel's ip= argument, and loopback when there is none. Code
inside the sandbox can still reach it, so this narrows what is exposed and is not
a boundary.

Signed-off-by: WaylandYang <145302500+WaylandYang@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…e status (#432)

The quick start stopped before the first sandbox and sent readers to
`fleet.sh seed` for a template, from before templates could come from an
image. It also promised a console that `cargo build` alone does not embed.
Build the console first, and end at a sandbox made from an image.

Node pairing and --tls-san move to docs/API.md, which already covers both.

Status said the backend moved to gVisor; Firecracker stayed as a second
backend (ADR 11). Parallel restores are done, so that item is ticked.

Signed-off-by: WaylandYang <145302500+WaylandYang@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
The Docker image no longer built: image templates compile in
rootfs-init, which the Dockerfile did not copy. Only a tag runs that
build, so nothing had noticed. The release binary, built on ubuntu-latest,
needed glibc 2.38 and would not start on Ubuntu 22.04 or Debian 12.

Build the binary on ubuntu-22.04, and add ci-release so both are tried on
a pull request rather than on a tag.

Track Cargo.lock, which was ignored, so a release builds what was tested,
and build with --locked.

Bump the core and SDKs to 0.6.0 and the MCP server to 0.3.0, and turn
Unreleased into v0.6.0 with the note that v0.5 does not upgrade in place.

Signed-off-by: WaylandYang <145302500+WaylandYang@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
api() read Firecracker's answer with a bare read, on a socket with a read
timeout. The kernel never restarts such a read, even for SA_RESTART, so an
interruption failed the call with "no answer" while Firecracker was still
working. Seen as a_load_repoints_the_disk_before_it_resumes and
a_failed_repoint_never_resumes failing about once in 40 runs of the lib
tests at 16 threads on 2 CPUs, with "Interrupted system call".

The new test signals the thread waiting for the answer, and fails without
this change in 5 of 5 runs. A kernel trace caught the same EINTR during a
run and the call carried on: 80 of 80 runs passed at that load. The trace
found no ordinary signal sent to the process at that moment, so what
interrupts the read is not known.

Signed-off-by: WaylandYang <145302500+WaylandYang@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: WaylandYang <145302500+WaylandYang@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
@WaylandYang
WaylandYang merged commit a9702d7 into main Nov 1, 2026
24 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants