Repository navigation
Release v0.6.0 - #437
Merged
Merged
Release v0.6.0#437
Conversation
After snapshot restore, the guest's KVM clock reflects the snapshot's saved time, not the host's current wall time. For snapshots created minutes or hours ago, the guest clock starts in the past, causing TLS certificate validation failures, incorrect timestamps, and token expiry issues inside the guest. This adds Phase 2.5 to restore_many_with: after the parallel restore PUT completes, poll each child's guest agent (forkd-agent.py on port 8888) until it responds, then exec `date -s @<host_unix_ts>` to set the guest clock to the host's current wall time. All children in a batch are synced in parallel. This is the cross-platform, cross-kernel-version alternative to the `clocksource=kvm-clock` boot arg (PR #289), which is x86-only and forward-only (requires re-baking existing snapshots). This approach works on any architecture, any Linux kernel, and any existing snapshot without re-baking. Best-effort and non-fatal: if the guest agent doesn't come up within 10 seconds, or `date` is unavailable, the sandbox remains usable with a stale clock. Failures are logged as warnings. References #288 (stale sandbox VM clock problem). Signed-off-by: jrimmer <jrimmer@users.noreply.github.com> Signed-off-by: jrimmer <jason@rimmer.net>
Signed-off-by: jrimmer <jason@rimmer.net>
Fix a bug in sync_guest_clocks: the exec args were ["-c", cmd] which causes subprocess.run to try executing a program named "-c" (fails with FileNotFoundError). Changed to ["sh", "-c", cmd] so the date command runs through the shell as intended. The bug was silently masked because clock sync is best-effort (failures logged as warnings, not propagated) and the clocksource=kvm-clock boot arg (#289) was correcting the clock on modern kernels. Add sync_guest_clocks_corrects_wrong_clock integration test: - Boots a fresh VM via Vm::boot - Deliberately sets the guest clock to 2001-09-09 (old_ts=1000000000) - Calls sync_guest_clocks on the single VM - Verifies the guest clock is corrected to host wall time (within 5s) - Verifies the corrected time is NOT the mock old time This isolates the scripting solution: even if KVM_CLOCK_REALTIME already corrected the clock, the test forces a wrong clock and verifies the sync function fixes it. The mock clock underneath is fine — we're testing the mechanism (date -s via guest agent exec), not the environment (kernel clock behavior). The test is #[ignore] (requires Linux + KVM + firecracker + rootfs) and #[cfg(target_os = "linux")]. Signed-off-by: jrimmer <jason@rimmer.net>
Three findings addressed: 1. Host timestamp captured before polling agent — a child that became ready near the 10-second deadline was set ~10 seconds in the past. Fix: capture SystemTime::now() per child AFTER ping succeeds, immediately before the date -s exec. This minimizes the gap between timestamp capture and clock correction. 2. restore_many_with unconditionally performed the agent protocol — custom/no-agent snapshots added up to 10 seconds to every restore even though failure is only a warning. Fix: clock sync is now opt-in via ForkOpts::sync_guest_clocks (default false). The daemon's spawn paths (create_sandbox, spawn_one_for_workspace) set it to true because all forkd images contain forkd-agent. CLI paths keep it false — the CLI doesn't know whether the image has an agent. 3. For n>1 with per_child_netns=false, every worker contacted the same 10.42.0.2:8888 — the shared IP/MAC topology is fundamentally conflicting. Fix: restore_many_with now rejects n>1 with per_child_netns=false with a clear error message directing callers to use per_child_netns=true for multi-child spawns. Signed-off-by: jrimmer <jason@rimmer.net>
sync_guest_clocks now returns (u128, Vec<ClockSyncOutcome>) instead
of just u128. ClockSyncOutcome is an enum with Synced{child_index}
and Failed{child_index, error} variants, exposed via a new
ForkResult::clock_sync_outcomes field.
Callers (e.g. the controller's create_sandbox) can inspect the
per-child outcomes to decide whether to fail-closed or degrade when
some children have unsynchronized clocks. The controller now logs
clock_sync_failures alongside the existing timing fields.
Also fixes the unused child_index warning from the thread closure
(the index is used in the collection loop, not inside the thread).
Tests:
- clock_sync_outcome_construction_and_matching: verifies enum
discriminants, field access, and the filter-for-failures pattern
callers use to decide fail-closed behavior.
- sync_guest_clocks_corrects_wrong_clock (#[ignore] integration):
now also asserts the outcome vec contains exactly one Synced
entry for the single child.
- fork_result_has_clock_sync_ms_field: updated to include the new
clock_sync_outcomes field.
Signed-off-by: jrimmer <jason@rimmer.net>
Two CI/coverage gaps from the review:
1. Clippy (rust CI was red): the sync_guest_clocks child loop used
.enumerate() and discarded the index (clippy::unused_enumerate_index).
Drop the .enumerate() and iterate children directly.
2. Behavioral coverage for the three review asks:
- multi-child address/mode: restore_many_with_rejects_multi_child_shared_ip
unit test proves n>1 with per_child_netns=false is rejected before any
Firecracker spawn or filesystem side effect.
- no-agent/disabled-sync latency: fork_opts_sync_guest_clocks_defaults_disabled
unit test proves the opt-in default keeps sync off (no added latency for
custom/no-agent images).
- delayed agent readiness (timestamp freshness):
sync_guest_clocks_delayed_agent_readiness (#[ignore] integration) boots a
VM, calls sync_guest_clocks immediately (no pre-wait) so it must poll for
the not-yet-ready agent, then asserts the guest clock matches host wall
time within 3s — a stale pre-poll timestamp would drift by the poll
duration.
Signed-off-by: jrimmer <jason@rimmer.net>
WaylandYang's final review round (round 5, 2026-08-14) flagged the
delayed-agent-readiness integration test as non-deterministic: it
called sync_guest_clocks immediately and only checked the final drift
plus an upper elapsed bound, so it never guaranteed that agent
readiness was actually delayed beyond the accepted 3-second drift. If
the agent came up quickly, the old pre-poll-timestamp implementation
still passed the test.
Extract the timestamp-placement logic from sync_guest_clocks into
sync_one_child_clock(ping, now, set_clock), which accepts injectable
readiness-probe / timestamp-source / clock-setter closures. The
production call site wires these to ping_at/ping_in_netns,
SystemTime::now, and the date -s exec; the test wires them to a
deterministic probe that fails for 4 seconds (> the 3s accepted drift)
then succeeds.
The new unit test sync_one_child_clock_timestamp_fresh_after_delayed_ping:
1. asserts the induced poll delay (4s) exceeds the accepted drift (3s),
so the freshness assertion is meaningful,
2. asserts the timestamp passed to set_clock was captured AFTER ping
succeeded (now_called_at >= probe_ready_at, freshness_gap < 3s) --
a pre-poll-timestamp implementation would fail by ~4s,
3. asserts set_clock was actually invoked with that fresh timestamp.
No VM, guest agent, or KVM is required -- it runs in the unit-test suite
on every platform. The end-to-end mechanism (real agent + date -s) is
still covered by the #[ignore] integration test
sync_guest_clocks_corrects_wrong_clock.
Verified on Linux (rust 1.83, x86_64): cargo clippy clean for
forkd-vmm/controller/cli/uffd; cargo test -p forkd-vmm --lib passes
48 passed, 0 failed, 3 ignored.
Signed-off-by: jrimmer <jason@rimmer.net>
Apply the three cross-reviewer findings from the ce-code-review pass
(5 reviewers: correctness, testing, maintainability, reliability,
adversarial):
1. Timeout error dropped the agent address (P2, safe_auto) — flagged by
correctness, testing, and adversarial. In multi-child restores the
operator could not tell which child's agent failed to come up. Now
wrapped at the call site: `.map_err(|e| e.context(format!(
"guest agent at {addr} did not come up")))`.
2. No unit test for the timeout path (P2, manual) — flagged by testing,
reliability, and adversarial. Only the #[ignore] integration tests
exercised it, and the 10s production timeout made a dedicated unit
test impractical. The timeout is now an injected parameter on
sync_one_child_clock, and sync_one_child_clock_times_out_when_probe_
never_succeeds uses a 500ms injected timeout to verify the bail
message and that now()/set_clock are never called on the timeout
path.
3. No unit test for set_clock failure propagation (P2, manual) —
flagged by testing, reliability, and adversarial.
sync_one_child_clock_propagates_set_clock_failure verifies a
set_clock error propagates verbatim and that now() was still called
(timestamp captured before the failed set).
Acknowledged (P3 advisory, no action): the 4s induced delay makes the
freshness test the slowest non-ignored unit test; wall-clock second
granularity in the freshness assertions is robust given the 4s>3s
headroom and the compile-time invariant guard.
Verified on Linux (rust 1.83, x86_64): cargo clippy clean for
forkd-vmm/controller/cli/uffd; cargo test -p forkd-vmm --lib passes
50 passed, 0 failed, 3 ignored.
Signed-off-by: jrimmer <jason@rimmer.net>
Signed-off-by: jrimmer <jason@rimmer.net>
fix(vmm): best-effort guest clock sync after snapshot restore (Option B)
reconcile() took K+4 separate registry-mutex acquisitions for a pass that prunes K stale sandboxes: one lock per removed id, plus one each for the stale scan, the live-id snapshot, and the workspace walk. Two handler calls interleaved between the scan and the per-id removal could make the reported prune count diverge from what was actually removed. Collect the stale ids, remove them, and walk the workspaces for the Running -> Stale transition all under one `self.inner.lock()` acquisition. `pid_alive` is a `/proc` stat and takes no lock we care about, so holding the registry mutex across it is fine. flush() behavior is unchanged: it still runs once, after the lock is dropped, only when something changed. Adds coverage for reconcile()'s prune-count and workspace-transition behavior (previously untested): a dead sandbox is pruned and its Running workspace flips to Stale with live_sandbox_id cleared; a sandbox with no recorded pid is left alone and its workspace stays Running; a Suspended workspace is never touched; and a no-op pass does not rewrite state.json. Fixes #310 Signed-off-by: Amir Fathi <amirfathi.me@gmail.com>
…ock-310 fix(controller): run Registry::reconcile under a single lock
Squash of 10 commits on fix/restore-path-resilience for rebasing onto current dev (which now contains #281's shared-TAP lifecycle). Original commits: - 858f63b fix(vmm): restore-path resilience for issue #301 - ce3d800 fix(vmm): apply review fixes for #301 - 0d73362 fix(vmm): make DEFAULT_SOCKET_WAIT_SECS pub for cross-crate use - e820848 fix(controller): use Vm::pid() accessor, not private field - 1ddf40c fix(controller): drop needless borrow in workspace orphan scan call - ca52820 fix(vmm): partial-success contract + concurrent socket wait for restore_many_with - 9e863de fix(controller): expose orphan-firecracker detection as a metric - 2aca840 test(vmm): integration test for partial-success restore - 9342e7a test(vmm): accept Restore or SocketWait phase in partial-success test - f2751b9 fix(vmm): review fixes for #301 — indexed success, valid test topology, docs Signed-off-by: jrimmer <jason@rimmer.net>
Signed-off-by: jrimmer <jason@rimmer.net>
Review #302: restore_many_with flattening children via children::flatten() renumbered ClockSyncOutcome.child_index, so a survivor past a failed hole (spawn/socket-wait) was reported as 0..n instead of its original 1-based within-batch slot, breaking the resource correlation that ForkChild/RestoreFailure promise. - sync_guest_clocks now takes &[(usize, &Vm)] where the usize is the original 1-based within-batch slot; outcomes echo that slot in both ClockSyncOutcome::Synced/Failed and the tracing child spans. - Add live_children_with_indices(): the single index-preserving mapping (enumerate + filter_map, i+1) used by restore_many_with, extracted so the invariant is unit-testable without KVM. - Regression: live_children_with_indices_preserves_survivor_slot_ across_hole proves [None,Some,None,Some] -> slots [2,4], not [0,1]. - Update the single-child clock-sync integration test to pass slot 1 and assert child_index == 1. Signed-off-by: jrimmer <jason@rimmer.net>
Map a line directly following the #Cons list was parsed as an unindented list continuation (-D warnings / clippy::doc-lazy-continuation). Separate it with a blank doc line. Signed-off-by: jrimmer <jason@rimmer.net>
fix(vmm): restore-path resilience — configurable timeout, per-VM partial-failure reporting, pre-restore orphan detection
Issue #298: when the controller crashes or restarts while Firecracker processes are still alive, the restarted controller cannot reconstruct resource ownership for orphaned VMs. Both the NetnsAllocator (active set) and shared_tap_owner are initialized empty, so a new spawn can reserve an orphaned VM's netns index or claim the shared tap even though the orphan still owns it. Fix: add `Registry::kill_orphans()` called after `reconcile()` on startup. After reconcile() prunes dead-PID entries, any remaining sandbox entries have alive PIDs but no live_vms handle — they are unmanageable orphans. kill_orphans() kills each orphan (after verifying the PID belongs to a Firecracker process via /proc/<pid>/comm to guard against PID reuse), prunes the registry entry, and marks workspaces whose sandbox was killed as Stale. After this, the NetnsAllocator (empty active set) and shared_tap_owner (None) are safe: no orphaned VM holds a netns index or the shared tap. Closes #298 Signed-off-by: jrimmer <jason@rimmer.net>
Address all P0 and P1 findings from the 7-reviewer ce-* panel: P0: kill_pid failure no longer prunes the registry entry. ESRCH (process already dead from TOCTOU race) is distinguished from real errors (EPERM etc.). On ESRCH, the entry is safely pruned. On real errors, the entry is kept in the registry to prevent the exact resource collision issue P1 fixes: - Wait for process death after SIGKILL (5s bounded timeout polling /proc/<pid> disappearance). A D-state process stuck on I/O can hold netns/tap resources past the kill return; if it doesn't exit within the timeout, the registry entry is kept. - pid_is_firecracker uses exact match (== "firecracker") instead of substring match to avoid false-positive kills of unrelated processes. - Separate KillOrphansResult struct (killed, pruned_stale, kill_failed) replaces the misleading single usize counter. The startup log now reports all three counts. - Extract mark_stale_workspaces() helper to eliminate the ~22-line duplicated workspace stale-marking block between reconcile() and kill_orphans(). - kill_orphans is pub(crate) with a doc invariant: "MUST only be called before any Vm handle is held" — prevents future post-startup misuse. - Defensive pid <= 1 guard in pid_is_firecracker (defense-in-depth against corrupted state.json). - ESRCH downgraded from error! to debug! (benign TOCTOU race). - SAFETY comment added on unsafe libc::kill. New tests: pid:None entries skipped, multiple workspaces same sandbox, persistence to disk after kill_orphans, reconcile+kill_orphans integration. Signed-off-by: jrimmer <jason@rimmer.net>
WaylandYang round 4 found that kill_orphans correctly keeps registry entries on EPERM/timeout (kill_failed > 0), but run_daemon only logs the count and continues. The NetnsAllocator active set and shared_tap_owner start empty, so the next spawn can reuse the failed orphan's netns index or tap lease — the exact collision #298 prevents. Fix: run_daemon now aborts startup when kill_failed > 0, forcing operator intervention rather than silently admitting conflicting spawns. The error message names the count and tells the operator to kill the orphans manually and restart. Also updated the pid_is_firecracker doc comment to be honest about the PID-reuse limitation: the comm check is a defense-in-depth heuristic, not complete PID-reuse protection. A robust fix would use pidfd signaling plus durable identity validation (process start time recorded with the registry entry at spawn time, compared on startup). That is tracked as a follow-up. Added test: kill_orphans_result_kill_failed_contract verifies the KillOrphansResult contract (kill_failed == 0 on empty registry, so baseline startup does NOT abort). Signed-off-by: jrimmer <jason@rimmer.net>
…decision The startup abort logic was inline in run_daemon, making it impossible to unit-test the kill_failed > 0 → bail decision path. Extracted into pub(crate) check_orphan_kill_result(&KillOrphansResult) -> Result<()> so the abort contract is directly testable. Tests: - check_orphan_kill_result_aborts_on_kill_failure: verifies that kill_failed=1 returns Err with "aborting startup" message, and kill_failed=0 returns Ok even with killed/pruned entries. - check_orphan_kill_result_error_contains_count: verifies the error message contains the exact kill_failed count so operators know how many orphans need manual intervention. The existing kill_orphans_result_kill_failed_contract test (zero-baseline) is retained as documentation of the baseline contract. The new tests exercise the actual abort decision logic. Signed-off-by: jrimmer <jason@rimmer.net>
The dead-last_err cleanup dropped the declaration and the unreachable expect, but the `last_err = Some(e)` assignment (guarded by the tap/cgroup-busy warn block on main) survived the rebase and referenced a now-missing binding. Remove it — the loop already returns on the final attempt, so the assignment is dead. Signed-off-by: jrimmer <jason@rimmer.net>
Replace the comm=="firecracker" check with durable process identity verification and pidfd signaling, addressing review r6 (WaylandYang) and the ce-code-review security findings. The comm check was spoofable (prctl PR_SET_NAME) and could not distinguish two Firecracker processes sharing a PID over time — a same-name PID-reuse attack would kill a legitimate new Firecracker, and on a multi-tenant host a cross-tenant DoS was possible. The TOCTOU window between the check and kill(2) was also acknowledged. Changes: 1. SandboxInfo::proc_starttime (serde default) — persists the process start time (field 22 of /proc/<pid>/stat, clock ticks since boot) captured at VM registration. Backward-compatible: old state.json files without this field deserialize to None. 2. read_proc_starttime — robust parse of /proc/<pid>/stat that handles the parenthesized comm field (field 2, can contain spaces/parens) by splitting at the last ')' and indexing field 22 in the remainder. 3. process_identity_matches — compares the live start time against the recorded one. Returns Match / PidReuse / Dead / Unknown. PidReuse means the original process exited and the PID was recycled — prune the stale entry without killing. Unknown (no recorded start time, /proc unreadable, or off-Linux) fails closed. 4. pidfd_open + pidfd_send_kill — signal through a pidfd opened AFTER the identity check, closing the TOCTOU window: the fd is pinned to the verified process, so a PID reuse between check and signal cannot redirect the kill. Uses raw libc::syscall with SYS_pidfd_open / SYS_pidfd_send_signal (Linux 5.3+; libc 0.2.x exposes the syscall numbers but not named fns). 5. comm_is_firecracker — retained as a secondary confirmation after the primary start-time check passes, to catch corrupted state.json. Not a security boundary. 6. kill_orphans reworked: Match → pidfd kill + wait_for_death; PidReuse → prune stale (no kill); Dead → prune stale; Unknown → fail closed (keep entry, increment kill_failed so run_daemon aborts startup). Tests updated to reflect the new fail-closed semantics: - kill_orphans_prunes_alive_pid_entries now uses a mismatched start time (u64::MAX) to trigger PidReuse → prune, not the old None path. - kill_orphans_fails_closed_on_unknown_identity (new): alive PID with proc_starttime: None → fail closed, entry kept, kill_failed=1. - kill_orphans_prunes_same_name_pid_reuse_without_killing (new regression): the same-name Firecracker PID-reuse attack the reviewer flagged — a mismatched start time must prune without killing, even if comm were "firecracker". - All other kill_orphans tests updated to use mismatched start times where they previously relied on the comm-check prune path. Verified on Linux (rust 1.83, x86_64) via Docker: clippy clean for forkd-controller; cargo test --all-features passes 104 unit + 8 integration tests, 0 failed. Signed-off-by: jrimmer <jason@rimmer.net>
Address ce-code-review findings (security P1, testing P1, adversarial P2/P3, plus the cross-reviewer coverage gaps) on the durable-identity rework from 2e275fe. 1. Residual TOCTOU window (security P1, adversarial ADV-1): pidfd closes only the open→signal window, NOT the check→open window. The original could exit and the PID could be reused by another Firecracker between process_identity_matches (which reads /proc/<pid>/stat) and pidfd_open. Fix: after pidfd_open succeeds, re-read the live start time and compare to the recorded one. A mismatch means the PID was reused between the check and the open → prune without killing (PidReuse). The pidfd pins the process at open time, so /proc/<pid> reflects the pinned process. This closes the check→open window; the comm check is now a pure sanity check, not a load-bearing defense. Design comment corrected to not overstate the guarantee. 2. Off-Linux comment/behavior mismatch (adversarial ADV-2): the module comment claimed off-Linux "prunes without killing" but Unknown actually fails closed (keeps entry, increments kill_failed, run_daemon aborts). Fixed the comment to state the real behavior. 3. Cross-platform test guards (adversarial ADV-3): the PidReuse- asserting tests (mismatched starttime → pruned_stale) now carry #[cfg(target_os = "linux")] because off-Linux returns Unknown (kill_failed), not PidReuse. The cross-platform fail-closed test (proc_starttime: None → Unknown) stays ungated. 4. Match-arm coverage (testing P1, +reliability/security/adversarial): new kill_orphans_match_arm_kills_verified_child_via_pidfd spawns a real child, records its true start time (Match), and asserts the secondary comm_is_firecracker check fails closed for a non-firecracker process — proving the Match arm is entered and the comm gate works as defense-in-depth. (Killing a real Firecracker would require a Firecracker binary; the comm gate is the testable boundary here.) 5. start-time parser unit test (testing P2): read_proc_starttime now has a direct test (parses our own /proc/<pid>/stat, rejects pid<=1, returns None for nonexistent PID). 6. Backward-compat serde test (testing safe_auto): new proc_starttime_defaults_to_none_when_absent_in_old_state_json deserializes a hand-written pre-r6 state.json (no proc_starttime key) and asserts #[serde(default)] yields None — the contract that lets existing deployments upgrade without wiping state. Verified on Linux (rust 1.83, x86_64) via Docker: clippy clean for forkd-controller; cargo test --all-features passes 107 unit + 8 integration tests, 0 failed (3 new tests pass). Signed-off-by: jrimmer <jason@rimmer.net>
…ding comments Review r8 (2026-08-20) found that kill_orphans_match_arm_kills_verified_child_via_pidfd did not do what its name claimed: it spawned 'sleep' (comm='sleep'), so the secondary comm_is_firecracker check failed closed and the irreversible success path (pidfd_send_kill -> wait_for_death -> registry removal) was never executed. - Rename the existing test to kill_orphans_match_arm_comm_mismatch_fails_closed so its name and doc accurately describe the comm-gate fail-closed branch it covers. - Add kill_orphans_match_arm_kills_verified_firecracker_via_pidfd, which spawns a disposable process whose /proc/<pid>/comm is literally 'firecracker' (a copy of 'sleep' renamed to 'firecracker'), records its true start time, and asserts killed == 1, kill_failed == 0, pruned_stale == 0, and the registry entry is removed — exercising the full irreversible pidfd kill path. Also correct two misleading comments flagged in the same review: - SandboxInfo::proc_starttime claimed legacy None state may use comm-only verification; the implementation fails closed (keeps the entry, increments kill_failed) and never does comm-only kill. - IdentityCheck::Unknown claimed the caller chooses between a comm-only kill or prune; kill_orphans keeps the entry and aborts startup via kill_failed. Signed-off-by: jrimmer <jason@rimmer.net>
Signed-off-by: jrimmer <jason@rimmer.net>
Addresses review #299 (both blockers) and rebases onto dev with #312's single-lock reconcile preserved. Blocker 1 — cross-reboot false Match (PID+starttick collision): proc_starttime is ticks since boot, only unique within one host boot. The registry persists across reboots, so after a reboot an unrelated Firecracker can share both the numeric PID and the boot-relative start tick of a recorded entry, and a bare starttime check would SIGKILL the wrong process. Persist /proc/sys/kernel/random/boot_id at registration (SandboxInfo.boot_id) and gate the identity check on it: - different boot id -> PidReuse (prune WITHOUT signaling; the old process cannot still exist across a reboot) - missing recorded boot id (legacy state.json) -> Unknown (fail closed) - live boot id unreadable -> Unknown (fail closed) New read_boot_id() helper; recorded at all three production VM registration sites in http.rs. Blocker 2 — pid: None entries recreated the #298 collision risk: Such rows were skipped by both reconcile and kill_orphans, so startup succeeded with an empty allocator/shared-tap ownership while a live VM may still hold those resources. kill_orphans now counts retained rows without a PID into KillOrphansResult.unresolved, and check_orphan_kill_result aborts startup on unresolved > 0 (fail closed). Rebase: preserved #312's single-lock reconcile (whole in-memory pass under one registry lock, including inline workspace-stale marking); kept mark_stale_workspaces only for kill_orphans' use after pruning. Tests: - kill_orphans_does_not_kill_across_boot_id_mismatch: matching PID + real start time but a DIFFERENT recorded boot id -> pruned without signaling (killed==0, pruned_stale==1). - kill_orphans_counts_pid_none_entries_as_unresolved (replaces kill_orphans_skips_pid_none_entries): pid:None rows surface as unresolved and block startup via check_orphan_kill_result. - check_orphan_kill_result_aborts_on_unresolved: startup-decision regression. - Existing kill-path tests set boot_id to the current boot id where they exercise the Match/PidReuse path; the legacy fail-closed test keeps boot_id: None. Signed-off-by: jrimmer <jason@rimmer.net>
The rebase onto post-#302 dev merged both sides of the busy-retry block textually but not semantically: in the Err(e) arm, last_err = Some(e) (and the final_attempt variant Some(e.context(...))) both move e, then the trailing tracing::warn!(error = %e, ...) borrowed the moved value. #302 had moved that warn into the else branch ahead of the assignment. The arm already computes let msg = format!("{e:#}") for busy-classification, so use error = %msg in the warn to avoid re-borrowing e after the moves.
…phan-reaping kill_orphans_match_arm_kills_verified_firecracker_via_pidfd spawned its firecracker-named 'sleep' via sh -c '<bin> 30 & echo $!' and relied on the backgrounded process being reparented to init, so wait_for_death would see /proc/<pid> disappear right after SIGKILL. On the GitHub Actions runner that detached child is reaped almost immediately (diagnostic showed state=Z ppid=1 at t=0ms, dead by 5ms), so process_identity_matches returned Dead, kill_orphans pruned it, and the test failed with pruned_stale=1 instead of killed=1. Spawn the process as a DIRECT child the test owns (the runner cannot reap an owned, non-orphaned child), and reap it via a small reaper thread that calls Child::wait(). wait_for_death (polling /proc/<pid>) then observes the SIGKILL reaping and the Match arm reports killed=1 as intended.
…p (review #299 r9) Legacy state.json rows (no recorded boot identity via serde defaults) took the generic Unknown path on first restart after upgrade and aborted startup with an EPERM-style message describing neither cause nor fix. - IdentityCheck::LegacyUnverifiable: missing recorded starttime OR boot id is reported distinctly from a genuinely unverifiable environment. - KillOrphansResult.legacy_unverified counts legacy rows separately from kill_failed; run_daemon logs include the new counter. - check_orphan_kill_result emits a dedicated upgrade diagnostic (kill the recorded PID manually or remove the entry once verified dead), distinct from the kill-failure and pid-less-row messages. - Off-Linux, identity can never be verified, so the gate skipped any registry row and aborted dev-box startup; the abort decision is now extracted into a pure cross-platform orphan_abort_reason(), the Linux gate fails closed on it, and the off-Linux gate logs the counts and allows startup (review #299 non-blocking). - Corrected the wrong pidfd comment: /proc/<pid> reflects whoever currently holds the PID; the post-open starttime re-read is what makes the reuse check safe. - kill_orphans folds its two entry lock acquisitions into one guard, matching the #312 single-lock invariant. Tests: legacy rows count as legacy_unverified (not kill_failed); abort distinctness (legacy vs kill-failure diagnostics); Linux-gated gate wiring test; existing abort-contract tests retargeted to the pure decision function so they run on every platform.
* docs: correct security policy for current runtime * docs: clarify sandbox security limitations * docs: retain gvisor isolation warning * docs: tighten the security policy Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: WaylandYang <145302500+WaylandYang@users.noreply.github.com> --------- Signed-off-by: WaylandYang <145302500+WaylandYang@users.noreply.github.com> Co-authored-by: WaylandYang <145302500+WaylandYang@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…#408) The `sandstorm` CLI and `sandstorm-controller` left the tree, but the recipes, Hub, Arch package, Kubernetes manifest, host scripts and docs that needed them stayed, and none can run. The MCP server's `create_snapshot` tool called a route that no longer exists. docs/API.md described that daemon: the wrong port and flags, routes that are gone, and none of the node, template or tier routes. It now follows the router. Signed-off-by: WaylandYang <145302500+WaylandYang@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* feat(sdk-python)!: drive Sandbox through the master's API Sandbox started sandboxes by running the `sandstorm` CLI, which left the tree in v0.6, so the package's default export failed on first use. It now creates through Controller, from a template or a tag, waits for a node to report the sandbox running, and sends the creation token on exec, eval and ping. Sandbox.connect wraps a sandbox that exists. eval returns what the snippet printed, since the master runs it through the template's eval_argv rather than a warmed interpreter. Controller gains create_sandbox, and exec and eval now wait for as long as the command may run rather than Controller.timeout. test_send.py and smoke-sdk.py went with the TCP path and result_json they tested. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(api): eval returns what the snippet printed The example showed a bare expression returning its value, but the master runs the snippet through eval_argv, so 2+2 prints nothing. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
When `runsc start` failed, for example on a template whose program is not in its rootfs, the container stayed stopped with its sandbox process alive in the cgroup. `observe` reported that as a running sandbox, so the master recorded an address, cleared the failure and never retried, and commands sent to it failed or timed out. Runsc now removes the container when create, start or restore fails, as Firecracker already stops its VM. The sandbox is then not observed, so it is retried and quarantined with `last_error` set. A failed `runsc create` on a leftover container also clears the way for the next attempt. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
The node removes a container that fails to start, so the sandbox is quarantined and Sandbox raises instead of returning. A program that starts and then exits is the case left. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
The tiers table stretched to full width on one long State cell, so its headers wrapped. Shorter cells let it fit, with memory and latency aligned. docs/OPTIMIZATIONS.md lists what was optimized for gVisor, for Firecracker and for both, with the before and after from when each change was made. Key features links to it. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…ck (#414) A sandbox its node put to disk to make room still had its REJECT in the host's OUTPUT when it came back, and that rule only goes once set_tier has finished. sync_clock reaches the agent through OUTPUT, so it spent its whole 2 s on refused connects and gave up, with the wake waiting behind it. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* bench: latency under gVisor, KVM and PVM on one build workloads.py asked runsc whether a sandbox had reached the disk, so under Firecracker it returned at once and measured hot sandboxes. It reads the sandbox's cgroup now, which works under either backend. The frozen wake in the lifecycle table was ~10 ms; measured through the gateway it is ~0.1 s. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * bench: PVM again, with counts that fit its host The first round asked 2.4 GB for 14 and 40 Firecracker guests, so everything waited on swap, and the column read as PVM being slow. With 8 and 20 it is not. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * bench: PVM's kept in memory, and what its 2 s wakes were Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
….0 (#420) Controller() went to http://127.0.0.1:8889, the v0.5 daemon's address. A master listens on https://127.0.0.1:8080. __version__ said 0.5.1 where the package says 0.5.3, and a test now keeps the two together. The package's description and keywords still called it a microVM primitive. Also drops stale lines: the ADR index said Firecracker had not run on KVM, and the sdk-python CI comment named tests that are gone. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* feat(node): start a batch's creates side by side A node started the sandboxes of a batch one after another: 45 ms each on Firecracker and 96 ms on gVisor, whatever the machine. A run of creates now goes one per CPU at once, each under its own sandbox's lock, and a create promises its memory as it is let through, so a burst is not judged against one reading of free memory. The namespace pool follows memory, 32 to 256: past it a burst waited on the RTNL lock, and 100 children took as long side by side as one after another. Running 100 at a time found two bugs that serial runs have too. A repeat of a finished destroy took down or deleted a veth by a name that was by then the pool's or another sandbox's, and a sandbox whose network never arrived stayed running where nothing could reach it, unretried. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * bench: why the gVisor create rows were slow Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* fix(node): a warm image does not outlive the host files under it On gVisor a sandbox runs on the host's /usr, bound in, and a warm image holds a program laid out for the files it had mapped. unattended-upgrades replaced python3.12 and libc under an 18-hour-old image, and every sandbox restored from it segfaulted 200 ms later. The node reported it gone, the master asked again, and the restore fell back to a cold boot: every create ran twice, and nothing said so. It read as warm creates being slow in a burst, 160 ms each against 37 ms for a fork. The key now follows when the bound directories last changed, one level down, so an upgrade seeds a new image. That cannot see everything, so a sandbox created again within 30 s of a warm start also drops the image. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * bench: a warm image under an upgraded host, before and after Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs: a line between two changelog entries Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
A snapshot's image was deleted as soon as runsc restore returned, before the program had run. With a library replaced under a snapshot, as unattended-upgrades does, the program then died, the master recreated the sandbox from its template, and the caller got a 200 from a sandbox with none of its memory. No error anywhere. The image now stays until the sandbox has lasted a minute. One that dies sooner reads as a snapshot again, fails with a reason, and gets a real try at most once a minute, so it comes back with its memory if the files do. A fork of a tag that dies the same way stops with that error rather than restarting for ever. The marker is on disk, so a node restart keeps it. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* fix(node): a sandbox waiting to retry costs no polls The master answers a poll at once if it has a command, and diff issues one for everything that has not converged, including what the node is backing off on. A node that dialled in skipped it and polled again straight away until the retry was due: about 350 polls a second and 40% of a core for one failing sandbox, with a master log line each time. The driver for a listening node already guards against this with same_batch. A node now says what it is putting off and for how long. The master leaves those out and parks the poll, and hands them over once they are due. Commands for other sandboxes still arrive at once. Forgetting the failures of a sandbox the master stopped asking about skips these, or their attempts would never count up to quarantine. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs: drop a comment about a return value that has gone Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
#425) a_restore_that_fails_removes_the_container failed on dev and on a PR, with calls "delete " where "restore delete " was expected: runsc never ran. The test wrote the script from this process, and a test forking on another thread keeps the write descriptor until it execs, so exec of the script failed with ETXTBSY. In the same shape on Linux, 176 of 32,000 execs failed. With a child writing the script, none did. The real runsc is not written by the process that runs it, so this is the test's own race. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
A closed listener lives on in any process another thread has just forked, until it reaches exec. Rebinding its port then fails with EADDRINUSE, and the sandbox is unreachable until the next report. The gateway tries again for up to 100 ms. Found through tests that failed about one full run in ten, which also turned the rust check on dev red. Two others had their own causes: - backend.rs ran a fake runsc it had just written, and a child forked by another test held the write descriptor: ETXTBSY. A child process installs it now. - a_fork_in_a_group_can_start_its_fetch fetched from an address refused at once, so the fetch could fail before the second fork asked. It now fetches from a listener that never answers. Signed-off-by: WaylandYang <145302500+WaylandYang@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
A sandbox could reach nothing beyond its node and nothing could change that. Each now has a policy: internet for public addresses, allow and deny for IPv4 addresses and CIDRs. The console sets it per sandbox and sets the default for new ones, and the API and the E2B front door take it. The default stays closed, so nothing changes until someone opens it. A sandbox with no exception costs no rule. A node installs exceptions per sandbox in one iptables-restore transaction and reports what it installed, so a restarted one gets them back by the usual diff. Outbound traffic with no exception is now refused rather than dropped, so a closed sandbox looking a name up fails in about 0.1 s, not 5 s. Under gVisor a rootfs with no usable resolver gets public ones in the sandbox's layer, as Firecracker guests already did. Signed-off-by: WaylandYang <145302500+WaylandYang@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
A template was files an operator put on every node by hand. Now a node pulls an image from its registry itself, with no Docker, and builds what its backend reads. The master asks every node, including ones that join later, and what the image says to run becomes the template. The console has a form for it and a table of how each node got on. A layer is not unpacked by tar: it guards an archive against itself, not against the layer before it, and one layer's link would let the next write anywhere on the host. Each path is resolved as a process chrooted into the rootfs would resolve it. Blobs must hash to their digest. Signed-off-by: WaylandYang <145302500+WaylandYang@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: WaylandYang <145302500+WaylandYang@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…rify it by name (#421) * fix(sdk)!: reach a master at its default address from TypeScript and MCP Both still defaulted to http://127.0.0.1:8889, the v0.5 daemon's address, and neither could trust the private CA a master signs its certificate with. They now default to https://127.0.0.1:8080 and read SANDSTORM_CA, as the Python Controller does, with no way to turn verification off. Node's global fetch has no CA option short of an undici dependency, so a CA goes through a small fetch over node:https. It states rejectUnauthorized, which otherwise follows NODE_TLS_REJECT_UNAUTHORIZED, and refuses a CA file with no certificate, since an empty `ca` means Node's public roots. CI ran only the MCP stdio test, so the new TLS tests would never have run. It now runs the whole directory. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(sdk)!: bring the TypeScript SDK and MCP server up to the v0.6 API Neither could start anything on a new master: both needed a snapshot tag, and a tag only comes from branching a sandbox. Both now start from a template, as the Python SDK does, and wait until a node reports the sandbox running, removing it if it never does. An exec sent earlier fails. What the master refuses or ignores is gone rather than advertised: diff, measure_diff, mode, wait, live_fork, hugepages and prewarm. Branch needs a tag. MCP no longer sends memory_limit_mib=256, which overrode every template's memory. eval returns what was printed. The TypeScript types describe the master's responses. Exec, eval and branch now wait as long as the master may take, where a flat 60 s timed out first. touch and list_templates are new in both, since a branch needs a hot sandbox. CodeQL flagged two test lines: the MCP stand-in allowed TLS 1.0, and the NODE_TLS_REJECT_UNAUTHORIZED test assigned the variable. The first now requires TLS 1.2. The second is what that test exists to check, and uses vi.stubEnv, which also restores it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix!: let clients verify a master by name, and close the remaining v0.6 gaps A master's certificate named only 127.0.0.1, localhost and the bind address, so a client on another machine that dialled a hostname was refused, and with verification not optional it had no way in. --tls-san adds names. 0.0.0.0 is no longer written into the certificate, since nobody can dial it, and a master bound to it with no --tls-san warns. Nodes pin the CA and are unaffected. A branch ignored mode="diff", so a caller got a full branch and was not told. Everything but "full" now answers 410, as docs/API.md said. The Python Controller drops what v0.6 refuses, as the other two clients did, needs a tag, and waits as long as a branch may take. docs/API.md gains GET /ca.pem. sandstormd.service and the Dockerfile move to port 8080, and the service keeps its token in pki/api-token like any other master. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix: keep the bind address on the certificate, and fix what review found The last commit took 0.0.0.0 off the master's certificate as a name nobody can dial. A node in the same process dials exactly that, and only its registration is pinned: every later call verifies the name. So `--master --node --bind 0.0.0.0` could not poll, and fleet.sh's curl to the bind address failed the same way. It stays. The help text and CHANGELOG said nodes do not need --tls-san, which was wrong for the same reason. --tls-san values are trimmed and blanks dropped. MCP spawn_sandboxes removes its sandboxes on any failure, since their tokens exist nowhere else, and its timeout covers the whole batch. delete_snapshot encodes the tag. A failed read of the CA file is not cached. eval errors carry the exit code, since the error text can be empty. Sandbox.create no longer lists every CreateOptions field. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* docs(contributing): name every branch feature/<short-name> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: WaylandYang <145302500+WaylandYang@users.noreply.github.com> * docs(contributing): apply the branch rule to features only Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: WaylandYang <145302500+WaylandYang@users.noreply.github.com> --------- Signed-off-by: WaylandYang <145302500+WaylandYang@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Rust takes about 5.5 minutes in CodeQL, nearly all of it extraction and queries over the whole crate, with no cache to use. 25 of the last 60 PRs touched nothing under crates/ or Cargo.*, and still waited for it. Default setup cannot filter by path, so this moves to workflows of our own. Rust runs on PRs that touch it, the other languages skip docs-only PRs, and a new push to a PR cancels the scan of the last one. Pushes to dev and main and the weekly run still scan everything. The categories match default setup's so alerts carry over. Default setup has to be turned off before these can upload. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
a_wake_waits_for_a_tier_change_in_progress starts eight wakes, but how many may wait depends on the machine's CPUs, so it failed on two. It now sets the limit it needs. Signed-off-by: WaylandYang <145302500+WaylandYang@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: WaylandYang <145302500+WaylandYang@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
The host reaches the agent at the address the guest was booted with, so it binds that, read from the kernel's ip= argument, and loopback when there is none. Code inside the sandbox can still reach it, so this narrows what is exposed and is not a boundary. Signed-off-by: WaylandYang <145302500+WaylandYang@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…e status (#432) The quick start stopped before the first sandbox and sent readers to `fleet.sh seed` for a template, from before templates could come from an image. It also promised a console that `cargo build` alone does not embed. Build the console first, and end at a sandbox made from an image. Node pairing and --tls-san move to docs/API.md, which already covers both. Status said the backend moved to gVisor; Firecracker stayed as a second backend (ADR 11). Parallel restores are done, so that item is ticked. Signed-off-by: WaylandYang <145302500+WaylandYang@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
The Docker image no longer built: image templates compile in rootfs-init, which the Dockerfile did not copy. Only a tag runs that build, so nothing had noticed. The release binary, built on ubuntu-latest, needed glibc 2.38 and would not start on Ubuntu 22.04 or Debian 12. Build the binary on ubuntu-22.04, and add ci-release so both are tried on a pull request rather than on a tag. Track Cargo.lock, which was ignored, so a release builds what was tested, and build with --locked. Bump the core and SDKs to 0.6.0 and the MCP server to 0.3.0, and turn Unreleased into v0.6.0 with the note that v0.5 does not upgrade in place. Signed-off-by: WaylandYang <145302500+WaylandYang@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
api() read Firecracker's answer with a bare read, on a socket with a read timeout. The kernel never restarts such a read, even for SA_RESTART, so an interruption failed the call with "no answer" while Firecracker was still working. Seen as a_load_repoints_the_disk_before_it_resumes and a_failed_repoint_never_resumes failing about once in 40 runs of the lib tests at 16 threads on 2 CPUs, with "Interrupted system call". The new test signals the thread waiting for the answer, and fails without this change in 5 of 5 runs. A kernel trace caught the same EINTR during a run and the call carried on: 80 of 80 runs passed at that load. The trace found no ordinary signal sent to the process at that moment, so what interrupts the read is not known. Signed-off-by: WaylandYang <145302500+WaylandYang@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: WaylandYang <145302500+WaylandYang@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Promotes
devtomainfor v0.6.0. Merge with a merge commit, not a squash, somainstays an ancestor ofdev.See CHANGELOG.md.
🤖 Generated with Claude Code