Skip to content

testing: unattended soak evidence for the bare engine #263

Description

@mfw78

Run the bare engine unattended and record what happened.

Why

docs/production.md publishes a systemd unit, six alert rules and a poison-recovery runbook, and there is no evidence any of it has run unattended. The only soak item reachable from this tracker was a cross-repo child pointing at shepherd, which is soak evidence for a different composition.

Nothing in this repository can run a soak today. There is no soak or load engine configuration and no just recipe for a long run; the justfile covers build, test, lint and ci only. engine.soak.toml and engine.load.toml were shepherd-side and never existed here. Writing that configuration is part of this issue, not a prerequisite of it.

The drain contradiction this issue originally carried is gone: #148 landed, and docs/production.md now documents [limits.shutdown] drain_secs defaulting to deadline_secs plus 30 seconds, with the forced exit 1 as deliberate. The readiness surface (#147) is still absent, but the handbook does not promise one, so the soak's job there is to say whether its absence is survivable rather than to catch a false claim.

Scope

  • Add a committed soak configuration for the bare engine and a just recipe that runs it, so the run is reproducible rather than one operator's shell history.
  • Run the bare nexum engine against a real chain for a sustained period with at least one module holding a resumable subscription, alongside at least one fixture from modules/fixtures (panic-bomb, fuel-bomb, memory-bomb, log-bomb, flaky-bomb, slow-host) to force the failure paths on demand rather than waiting for them.
  • Exercise and record: a clean SIGTERM drain within drain_secs, a drain that overruns and forces exit 1, a restart with cursor resume, a provider outage and reconnect, a module trap and its restart, and a poison quarantine.
  • Capture the metric series across the run, so the six alert rules can be checked against real data rather than assumed thresholds.

Open decisions

  • Which chain and RPC provider, and who pays for the sustained connection.
  • Duration, and what counts as sufficient. Pick a number rather than leaving "sustained" unfalsifiable.
  • Where the evidence lives: a docs/ run report, an issue comment, or a checked-in metrics dump. This decides whether the done-condition is auditable later.
  • Whether the run is manual and one-off before v1, or a scheduled workflow. Manual is the honest v1 scope.

Done when

  • The soak configuration and its recipe are in the repository.
  • There is a recorded run: its duration, what was exercised, the metric series, and what docs/production.md got wrong.

Sequencing

Soft-blocked by #147: a soak without a readiness surface cannot answer whether one is needed. Informed by #280, whose alert is one of the six being validated and which the provider-outage leg exercises directly. #146 determines what the metric capture can contain, so a run before it measures a thinner series. This is the only planned evidence that the published operator handbook is true, so it should precede #262 and the v1 tag.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    testingTest coverage, harness, mocks, or test-infrastructure work

    Type

    No type

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions