Run the bare engine unattended and record what happened.
Why
docs/production.md publishes a systemd unit, six alert rules and a poison-recovery runbook, and there is no evidence any of it has run unattended. The only soak item reachable from this tracker was a cross-repo child pointing at shepherd, which is soak evidence for a different composition.
Nothing in this repository can run a soak today. There is no soak or load engine configuration and no just recipe for a long run; the justfile covers build, test, lint and ci only. engine.soak.toml and engine.load.toml were shepherd-side and never existed here. Writing that configuration is part of this issue, not a prerequisite of it.
The drain contradiction this issue originally carried is gone: #148 landed, and docs/production.md now documents [limits.shutdown] drain_secs defaulting to deadline_secs plus 30 seconds, with the forced exit 1 as deliberate. The readiness surface (#147) is still absent, but the handbook does not promise one, so the soak's job there is to say whether its absence is survivable rather than to catch a false claim.
Scope
- Add a committed soak configuration for the bare engine and a
just recipe that runs it, so the run is reproducible rather than one operator's shell history.
- Run the bare
nexum engine against a real chain for a sustained period with at least one module holding a resumable subscription, alongside at least one fixture from modules/fixtures (panic-bomb, fuel-bomb, memory-bomb, log-bomb, flaky-bomb, slow-host) to force the failure paths on demand rather than waiting for them.
- Exercise and record: a clean SIGTERM drain within
drain_secs, a drain that overruns and forces exit 1, a restart with cursor resume, a provider outage and reconnect, a module trap and its restart, and a poison quarantine.
- Capture the metric series across the run, so the six alert rules can be checked against real data rather than assumed thresholds.
Open decisions
- Which chain and RPC provider, and who pays for the sustained connection.
- Duration, and what counts as sufficient. Pick a number rather than leaving "sustained" unfalsifiable.
- Where the evidence lives: a
docs/ run report, an issue comment, or a checked-in metrics dump. This decides whether the done-condition is auditable later.
- Whether the run is manual and one-off before v1, or a scheduled workflow. Manual is the honest v1 scope.
Done when
- The soak configuration and its recipe are in the repository.
- There is a recorded run: its duration, what was exercised, the metric series, and what
docs/production.md got wrong.
Sequencing
Soft-blocked by #147: a soak without a readiness surface cannot answer whether one is needed. Informed by #280, whose alert is one of the six being validated and which the provider-outage leg exercises directly. #146 determines what the metric capture can contain, so a run before it measures a thinner series. This is the only planned evidence that the published operator handbook is true, so it should precede #262 and the v1 tag.
Run the bare engine unattended and record what happened.
Why
docs/production.mdpublishes a systemd unit, six alert rules and a poison-recovery runbook, and there is no evidence any of it has run unattended. The only soak item reachable from this tracker was a cross-repo child pointing at shepherd, which is soak evidence for a different composition.Nothing in this repository can run a soak today. There is no soak or load engine configuration and no
justrecipe for a long run; the justfile covers build, test, lint andcionly.engine.soak.tomlandengine.load.tomlwere shepherd-side and never existed here. Writing that configuration is part of this issue, not a prerequisite of it.The drain contradiction this issue originally carried is gone: #148 landed, and
docs/production.mdnow documents[limits.shutdown] drain_secsdefaulting todeadline_secsplus 30 seconds, with the forced exit 1 as deliberate. The readiness surface (#147) is still absent, but the handbook does not promise one, so the soak's job there is to say whether its absence is survivable rather than to catch a false claim.Scope
justrecipe that runs it, so the run is reproducible rather than one operator's shell history.nexumengine against a real chain for a sustained period with at least one module holding a resumable subscription, alongside at least one fixture frommodules/fixtures(panic-bomb,fuel-bomb,memory-bomb,log-bomb,flaky-bomb,slow-host) to force the failure paths on demand rather than waiting for them.drain_secs, a drain that overruns and forces exit 1, a restart with cursor resume, a provider outage and reconnect, a module trap and its restart, and a poison quarantine.Open decisions
docs/run report, an issue comment, or a checked-in metrics dump. This decides whether the done-condition is auditable later.Done when
docs/production.mdgot wrong.Sequencing
Soft-blocked by #147: a soak without a readiness surface cannot answer whether one is needed. Informed by #280, whose alert is one of the six being validated and which the provider-outage leg exercises directly. #146 determines what the metric capture can contain, so a run before it measures a thinner series. This is the only planned evidence that the published operator handbook is true, so it should precede #262 and the v1 tag.