P4: run E11 with a real model in the loop instead of synthetic Execs

[pdettori]

Summary

E11's density/throughput numbers (knee at c=8, ~39-43 execs/sec) were driven by a synthetic "closed-loop-per-slot" model issuing trivial Execs (true, cat >/dev/null) back-to-back with no think-time gap and no real tool-call payload. This issue is the full validation experiment: replace the synthetic driver with a real agent loop (real model calls, real tool calls) and see whether the throughput/latency picture — and the "sandbox overhead hides behind think-time" argument — survives contact with realistic traffic.

Why this matters

• This is the deck's own flagged gap ("No model in the loop... the arithmetic of hiding replenishment behind thinking time is argued, not measured"), and it's the most expensive item in this epic to stand up, so it should come after the cheaper, more targeted items (P4: measure real model think-time distribution to size the sandbox latency budget #257 think-time distribution, P4: decompose resume (28.58 ms) and destroy (21.31 ms) into their sub-costs #258 resume/destroy decomposition) rather than before them — those inform what this experiment should even look for.
• Everything currently measured (E10's 52.74 ms, E11's throughput ladder) is a mechanism measurement, isolated from real workload shape. This is what turns it into a service-level measurement.
• It also tests whether the current synthetic-Exec shape (near-instant commands) has been hiding any latency that real tool calls (longer-running commands, larger outputs, more filesystem activity) would expose — e.g., mount-at-acquire cost, or Σ PSS growth under real memory usage rather than idle standbys.

What's already known / where to look

• Driver: deploy/microvm/e11-density.sh, drivingModel: "closed-loop-per-slot" — the code path that would need a real alternative. The script already supports a model-stub hook: SH_E11_MODEL_STUB_CMD (see deploy/microvm/EXPERIMENTS.md's "Model-stub dependency gap" note — "absent one it drives the mix itself directly, a disclosed stand-in for the real stub, not the stub itself").
• The harness (harness/) and Pi coding agent (pi-fork/) already run real agent sessions elsewhere in the repo; the P6 density work used a model stub with a declared latency profile (300ms TTFT, 12ms/token) rather than a live model — a middle ground between fully synthetic and fully real that's worth considering as a cheaper intermediate step.
• Open item from EXPERIMENTS.md's "what happens next" list: "Live gate at low concurrency — validation, not a headline number... blocked on: a model key and a deployed relay." This issue is effectively that item, scaled up from a validation gate to a full density re-run.

Open questions for the spec

1. Real model (which one, at what concurrency-affecting cost — e.g., a hosted API's own rate limits could become a confound), or a declared-latency stub calibrated to P4: measure real model think-time distribution to size the sandbox latency budget #257's measured think-time distribution (cheaper, and directly answers "does the mechanism hold under a realistic gap" without needing a live model dependency)? I'd lean toward the stub as a first cut, gated on P4: measure real model think-time distribution to size the sandbox latency budget #257 actually producing a distribution to calibrate it against.
2. Should this reuse the existing E11 ladder shape (sweep c, look for a knee), or does a real workload's arrival pattern (bursty, not closed-loop) need a different driving model entirely?
3. What tool-call mix is representative — the repo already has a git-chain-based tool workload from the P6 work that could be reused rather than invented from scratch.

Suggested scope for a first cut

Depends on #257 landing first (calibration data). Once available: repeat the c=1,2,4,8 ladder with a declared-latency model stub matching the measured think-time distribution, and check whether the knee position and throughput ceiling change materially from the fully-synthetic baseline.

---

Part of #256 (P4 epic: next experiments after the metal STOP).

[pdettori]

Epic: rossoctl/rossoctl#2546 (Stream 2 — microVM sandbox tier, via sub-epic #256)

Status update (2026-09-17)

Still open, no PR yet. E11's driver disclosed this gap itself — absent an
SH_E11_MODEL_STUB_CMD, e11-density.sh drives the Exec mix directly as "a disclosed
stand-in for P6 §5.4's model stub, not the stub itself." This issue is what closes that
gap, and depends on #257 (real model think-time distribution)
to size a realistic mix rather than a synthetic one.

Assisted-By: Claude (Anthropic AI) noreply@anthropic.com