P4: test the admission gate directly — sealed prediction 1 cannot be scored by sweeping slots (memory 1%, CPU 66-70% at 256 slots)

[pdettori]

Summary

Sealed prediction 1 ("Replenishment binds on process/memory count before CPU") is currently INCONCLUSIVE — its scorer needs a threshold crossing on memory or process count, and the metal run never produced one: hostCpuFraction stayed ~0.001 flat and Σ PSS never exceeded 0.41 GB, even at 128 resident microVMs (c=64, the top of the tested ladder). What was observed (CPU demonstrably never bound, knee is replenishment-bound) is consistent with the prediction's direction but is explicitly not a scored result per deploy/microvm/predictions.json §7.4's falsification criterion.

Why this matters (and why it's ranked last)

• This is a completeness item, not a throughput/latency blocker: nothing measured so far suggests memory or process count binds before replenishment does, and the stated goal (throughput-optimized sandbox service) doesn't obviously need this scored to make near-term engineering decisions. It matters for long-horizon capacity planning at extreme density, and for closing out the sealed prediction set honestly (2 supported / 0 falsified / 2 inconclusive / 1 not evaluable — this is one of the two inconclusive ones).
• On a 754 GiB, 72-cpu host, reaching an actual memory/process ceiling at the current ~3 MB PSS/VM unit cost would require a very large sweep (extrapolated back-of-envelope: on the order of 10^5 resident VMs by memory alone) — this is a long-running, resource-intensive experiment, which is the other reason it's ranked last.

What's already known / where to look

• deploy/microvm/predictions.json — prediction 1's exact wording and falsification criterion ("falsifiedBy": "CPU saturating at or before the memory gate first refuses"). Do not edit this file — it's SHA-256 pinned and enforced by pnpm -C experiments exec vitest run microvm-predictions; it was sealed before the first rung specifically so predictions can't be adjusted to fit results.
• Scoring logic: analyzeLadder in experiments/src/microvm-density.ts (detectKnee internally) — it scores structurally from a RungSample ladder's own fields (hostCpuFraction, pssBytes, processCount, etc.), so a new sweep just needs to produce records with a real crossing in one of those fields for the existing scorer to pick it up; no scorer changes should be needed.
• Admission control: remote-worker/internal/vmpool/config.go's MaxCommittedBytes/MemoryReserveBytes gate — this is the actual thing that would refuse admission before a real OOM, and it's configured, not derived from the host's physical RAM (flagged elsewhere as a value that can be "configured inert" if sized wrong for the host). Scoring this prediction meaningfully requires either raising this budget toward the host's real ceiling and sweeping until something gives, or deliberately sizing it low enough to force a crossing on purpose (a different, and arguably less interesting, experiment).

Open questions for the spec

1. Is the goal to find where this host would actually run out (raise MaxCommittedBytes toward the host's real ceiling and sweep c/D until something gives), or to test the admission-control gate itself (deliberately undersize the budget and confirm it refuses cleanly rather than free-running into an OOM)? These are different experiments with the same prediction as a side effect.
2. Given the scale involved (potentially tens of thousands of resident VMs to find a real ceiling), is there a cheaper proxy — e.g., artificially inflating VMOverheadBytes or GuestRAMBytes to force a crossing at a much smaller VM count, trading realism for a tractable sweep size?
3. Does pid_max (4194304, per the E10 rung records' limits field) or another kernel limit bind before memory does at any tractable scale, and is that itself worth knowing even if it's not what the prediction asks about?

Suggested scope for a first cut

A design decision before any run: pick the "test the gate" framing (open question 1) as the tractable first cut, since it's a bounded experiment (deliberately undersized budget, confirm clean refusal) rather than an open-ended search for a real-hardware ceiling.

---

Part of #256 (P4 epic: next experiments after the metal STOP).

[pdettori]

Epic: rossoctl/rossoctl#2546 (Stream 2 — microVM sandbox tier, via sub-epic #256)

Status update (2026-09-17)

Still open, no PR yet. E11 scored sealed prediction 1 ("replenishment binds on
process/memory count before CPU") inconclusive, not falsified — its scorer needs a
threshold crossing on memory or process count, and the 72-cpu/754GiB metal host never
produced one (Σ PSS stayed under 0.41GB at 128 microVMs). This issue is exactly the
ladder-to-an-actual-ceiling that would make the prediction scorable; it's the one open
item from E11 with a precisely named gap rather than a general follow-up.

Assisted-By: Claude (Anthropic AI) noreply@anthropic.com

[pdettori]

Not reachable by sweeping slots on this host — re-scoping to the "test the gate" framing

This issue's own back-of-envelope ("on the order of 10^5 resident VMs by memory alone") is now confirmed as the right order, which means the first of its two framings is not a tractable experiment here. Re-scoping to the second, which it already identified as the tractable first cut.

What the campaign measured, and why it settles the feasibility question

The ladder has since been pushed far past the c=64 this issue describes as the top of the tested range, with the slot cap raised so slots actually take effect (#305):

slots	Exec/s	host CPU	memory used of 754.5 GiB
64	577.55	66.3%	8.3 GiB
128	524.66	70.0%	8.7 GiB
256	417.24	68.3%	~8.7 GiB

Neither gate is approached, and throughput is falling before either could be. Concretely:

• Memory is not close. memAvailable never dropped below 745.7 GiB. Sampling 306 per-VM cgroups found memory.current at 0.9 MiB mean / 2 MiB max against a PerVMBytes limit of 288 MiB, memory.events all zero, and nothing within 8 MiB of the limit. Guest memory is one MAP_SHARED mapping of the snapshot memfile which the worker pins once via PinMemoryFile, so VMs share a single copy — the per-VM unit cost is ~1 MiB, not the ~3 MB this issue assumed, making the required VM count larger than its estimate.
• CPU never saturates either, sitting flat at 66–70% across a 4x slot range. So the falsification criterion ("CPU saturating at or before the memory gate first refuses") also cannot fire.
• Throughput peaks at 64 slots and declines, because the binding constraint is the pool's replenishment rate. Past the peak, adding slots lowers throughput, so a sweep toward a memory ceiling would be driving a system that is getting worse, and the VMs needed to reach 10^5 resident would never be concurrently live.

So prediction 1 stays INCONCLUSIVE and cannot be scored by extending this ladder. That is a finding rather than a gap: nothing observed suggests memory or process count binds before replenishment, and the campaign has now identified what does bind (see #307 and the sockwait issue).

Re-scoped ask

Take open question 1's second framing — test the admission-control gate itself: deliberately undersize MaxCommittedBytes, confirm it refuses cleanly rather than free-running into an OOM, and confirm the refusal is observable rather than silently converting into cold acquires. That is bounded, cheap, and tests something that matters operationally, given that SH_MAX_COMMITTED_MB is configured rather than derived and was measured as conservative by 144x against the worst VM observed.

Note for whoever runs it: a refused refill converts into a cold acquire by design, so it corrupts the cold metric on the same rung. Read coldAcquireRateTrue, not the proxy (#306).

Open question 3 (pid_max, 4194304) is untouched and still worth knowing. Open question 2's cheaper-proxy idea — inflating VMOverheadBytes/GuestRAMBytes to force a crossing at a small VM count — would score the prediction only in a configuration chosen to make it fire, which is worth stating explicitly if anyone proposes it as a substitute.

deploy/microvm/predictions.json remains SHA-256 pinned and must not be edited.