E11: replace grpcurl with a persistent-connection Go client (issue #291 item 3, now mandatory)

[pdettori]

Issue #291 deferred its item 3 — replacing grpcurl with a persistent-connection Go client — on the explicit grounds that committing to it before measuring would be a guess. The design doc set the decision criterion:

The control arm added here measures how much per-Exec cost grpcurl still contributes, and that number decides whether item 3 is needed before an authoritative metal run.

That measurement now exists, and the answer is yes.

The evidence

Measured on the bare-metal rig (srv-r16b14s16, 72 cpu / 754 GiB, virt: none, governor performance, gawk) with PR #293's driver, SH_E11_ARMS=driver-control, ITERS_PER_SLOT=200, SAMPLE_INTERVAL_MS=250, full published ladder. The driver-control arm drives the identical run_density_rung against cmd/null-responder: no relay, no Redis, no worker, no VMM, and a server that answers every Exec with one End and executes nothing. Every millisecond and every cycle below is driver overhead.

c	tput/s	p95	hostCpuFraction	hostCpuFractionPeak	coresBusy /72	hostCpuSamples
1	56.8	17 ms	0.0204	0.0253	1.47	15
2	101.3	19 ms	0.0411	0.0470	2.96	17
4	171.6	25 ms	0.0879	0.1050	6.33	20
8	241.6	44 ms	0.2017	0.2131	14.52	25
16	219.6	137 ms	0.4163	0.4864	29.97	46
32	204.9	340 ms	0.7564	0.8069	54.46	72
64	231.6	753 ms	0.8907	0.9598	64.13	103

Three things follow.

1. The driver alone has a knee at c=8. Throughput peaks there and then declines while p95 grows 44×. That is the same knee position and the same curve shape EXPERIMENTS.md §E11 published for both real arms — an arm with no backend at all reproduces the published result's shape.

2. The driver alone saturates a 72-cpu host. At c=64 it burns 64 of 72 cores and its CPU peak reaches 0.9598, crossing the 0.9 threshold crosses('cpu') tests. A sweep cannot attribute a resource bound to a backend when the load generator is consuming the resource.

3. The driver's own p95 is ~45% of the published microVM arm's. 753 ms of the published 1686 ms at c=64 is driver cost, before the backend does anything.

Corroboration that the arm is measuring what it claims: postLoadHostCpuFraction sits at 0.0006–0.0010 at every rung — i.e. the pre-#291 sampling method reproduces §E11's published "0.001 flat across the entire ladder" exactly, while the under-load mean climbs to 0.89.

Why grpcurl is the cost

One process per Exec, and each one re-parses the proto descriptor set and opens a fresh TCP connection and HTTP/2 session before sending a single request. PR #293 removed the ~8 other spawns per Exec (two of them Python interpreters) and the remaining per-Exec cost is one execve — this one.

What to build

A small Go client against the existing gen/go/sandbox/v1 stubs — no new codegen, the same package cmd/null-responder already uses:

• one process for the whole rung, one grpc.ClientConn reused across every Exec;
• c goroutines in place of c bash subshells, each with its own disjoint req_id range (see slot_req_base — two concurrent Execs sharing a req_id once hung a run for 33 minutes);
• host-side timestamps around each Exec call, emitted in the <ms> <status> <cause> line format grpc_exec_record already writes, so run_density_rung's aggregation, percentiles and record writer are untouched;
• opt-in behind an env var so the bash path stays the reference until the two are compared on the same host.

This also makes an open-loop / rate-based driver cheap, which the design doc records as viable only if item 3 is built. The current drivingModel is closed-loop-per-slot with a declared coordinated-omission bias that understates latency at saturation — exactly the regime prediction 3 lives in.

Acceptance

• The driver-control arm re-run with the Go client shows materially lower coresBusy and p95 at high c than the table above; if it does not, the bottleneck is elsewhere and that is itself the finding.
• Both drivers run against the null-responder on the same host, and the difference quantifies what the bash path cost.
• Only then is an authoritative three-arm sweep worth booking, and §E11's c=8 knee re-testable.

Blocks

The authoritative re-run in #291. Until this lands, a sweep would measure the driver's knee again — the caveat now in EXPERIMENTS.md §E11 stays in place, and §E11's conclusions stay marked as under repair.

🤖 Generated with Claude Code