diff --git a/openspec/changes/network-monitor-liveness-tests/.openspec.yaml b/openspec/changes/network-monitor-liveness-tests/.openspec.yaml new file mode 100644 index 00000000000..1b062d3a7c7 --- /dev/null +++ b/openspec/changes/network-monitor-liveness-tests/.openspec.yaml @@ -0,0 +1,2 @@ +schema: spec-driven +created: 2026-08-04 diff --git a/openspec/changes/network-monitor-liveness-tests/design.md b/openspec/changes/network-monitor-liveness-tests/design.md new file mode 100644 index 00000000000..995a330eed2 --- /dev/null +++ b/openspec/changes/network-monitor-liveness-tests/design.md @@ -0,0 +1,223 @@ +## Context + +Network monitor v3 today performs exactly one kind of measurement: a high-rate two-hop stress test against a single mixnode, routed `[tested_node, agent]` so the node relays each packet straight back. The orchestrator assigns work lazily from a staleness-ordered node table guarded by a `testrun_in_progress` lock set, agents are one-shot jobs that test exactly one node and exit, and results are submitted to nym-api in signed monotonic batches that feed the stress component of node performance. + +Liveness scoring meanwhile lives in the nym-api-internal v1 monitor, which measures delivery over five-node routes and attributes the result to one substituted node. Its bias is documented in its own design record and is the motivation for this change (see proposal.md). + +The relevant constraints are: + +- **The `network-monitors` contract cannot be redeployed, but it CAN be migrated.** It carries a `migrate` entry point guarded by cw2 `ensure_from_older_version`, and its admin is the Nymtech SA multisig, so an additive change is a governance action rather than a redeploy. The binding constraint is the third-party node fleet, and it is a constraint on the SHAPE of the change rather than on whether one happens: `cw_serde` does not set `deny_unknown_fields`, so an un-upgraded node silently ignores an unknown field on a message or query response whose variant it already knows, whereas a NEW or RETYPED `ExecuteMsg` variant fails to deserialise and the node's event handler logs and continues, which means it silently stops learning about agents. Additive optional fields are therefore fleet-safe; new or retyped variants are not, and this change uses only the former (Decision 14). +- **Agent and orchestrator redeploy together**, so wire changes between them are free. nym-node and nym-api do not, so anything they must learn is a compatibility surface. +- **Nodes learn the authorised-agent set only through a nyxd websocket event subscription** with no periodic reconciliation. A node that misses an event only re-syncs on restart. This is the load-bearing risk for a liveness score, because a node that has not ingested its agents' authorisations fails every gate and is indistinguishable from a dead node. +- **Node IP addresses are unique across the node population** (nodes attribute each other's Noise keys by IP), whereas several agents may share one IP disambiguated by port. +- Announced address sets are canonicalised, deduplicated and sorted, and test runs rotate through them by position. + +## Goals / Non-Goals + +**Goals:** + +- Produce a per-node liveness score that depends on that node and the probing agent's own link, and on nothing else in the network. +- Cover a gateway's two real jobs: forwarding client traffic into the mixnet, and delivering mixnet traffic to a client. +- Sweep the whole testable population at a cadence comparable to v1's 15 minutes. +- Ship in a way that cannot damage rewarding while the node and gateway fleets are still upgrading, and produce the evidence needed to decide when liveness may replace the v1 routing score. +- Leave the existing stress test's behaviour and scores unchanged. + +**Non-Goals:** + +- Removing or altering the v1 monitor. It keeps running and keeps feeding the routing component; the cutover is a later change gated on the evidence this one produces. +- Testing a gateway's wss / hostname ingress path. Recorded as a future test kind. +- Testing wireguard, SOCKS5, the authenticator, or anything else `nym-gateway-probe` and the node-status agent cover. +- Latency-derived scoring. Latency is recorded, not scored (Decision 11). +- Changing the source-IP basis of the node-side MIXNET gates (Noise responder, routing filter, replay bypass). Moving those onto the Noise-authenticated x25519 static key remains the separately-identified follow-on. This change re-keys only the new client-session gate, which no Noise handshake can ever cover (Decision 14). +- Removing agent IP addresses from the contract. Considered and rejected in Decision 14, because the node must know an agent's address to route the mixnode probe's return hop at all. +- Removing gateway on-disk message storage for ordinary clients. + +## Decisions + +### Decision 1: Liveness is a second test kind inside the same subsystem + +**Choice.** Liveness is a `test_kind` handled by the same orchestrator, the same agents, the same chain authorisation, and the same submission path as stress testing. Work becomes keyed by `(node_id, test_kind)` rather than by `node_id`. + +**Why.** Every actor, credential and lifecycle stage is already in place; the only genuinely new things are the probe profile, the gateway phases, and per-kind scheduling. A separate subsystem would duplicate the authorisation model, the node registry, and the submission machinery for no gain. + +**Alternative considered.** A distinct capability and daemon. Rejected as duplication. + +**Consequence.** Several structures that assume one test per node need widening (the staleness pointer, the rotation cursor, the submission watermark), and the assignment API grows a kind discriminator. The `testrun_in_progress` primary key on `node_id` alone is deliberately NOT widened: it is what keeps two kinds from measuring one node simultaneously. + +### Decision 2: Mixnode liveness is the existing probe at a low-volume profile, still replaying one header + +**Choice.** The mixnode liveness probe is the existing two-hop self-loop with its own profile (order 100 packets, lower rate, shorter straggler wait), keeping `reuse_header` enabled. + +**Why.** The measurement being replaced is a delivery ratio, and the existing probe already produces one with correct attribution. Keeping `reuse_header` keeps a single code path through the node (Noise responder, routing filter, replay bypass, forward hop) so a liveness result and a stress result describe the same machinery, and it avoids rebuilding a header per packet across a whole wave. + +**Alternative considered.** Fresh headers per packet, which would decouple liveness from the replay/bloomfilter bypass and make wave attribution trivial (Decision 6). Rejected because it introduces no new independence in practice: the bypass is gated on the same authorised-agent set as the Noise and routing gates, so a node that fails one fails all three. + +**Consequence.** A node whose chain subscriber is broken fails liveness for a reason unrelated to its forwarding capability. That is the central risk of the change and is handled by shipping at weight zero (Decision 12), not by the probe design. + +### Decision 3: Gateway liveness is one indivisible test with two phases over one client session + +**Choice.** A gateway liveness assignment is a single unit of work performed by a single agent within a single client session, in two phases: + +``` +phase 1, ingress phase 2, egress +agent(client) --ws:ForwardSphinx--> GW agent(as mixnode) --mixnet--> GW + GW forwards verbatim to next hop GW: Noise responder, final-hop unwrap, + --> agent's own mixnet listener destination resolution + --> push into the live client session +tests: session, bandwidth path, tests: mixnet ingress, sphinx unwrap, + outbound forwarder + Noise client delivery +``` + +The run produces two signals. The score denominator is fixed by the kind at two signals, so a phase that produces nothing scores zero rather than being dropped from the average. A phase-1 failure MUST NOT abort the run. Only failure to establish the session aborts, in which case both signals are zero. + +**Why.** The two phases test independent capabilities and a gateway needs both to be useful, so both must always be measured and a missing signal must never be more favourable than a zero one. This mirrors v1, which seeds every tested node at zero received so that an unreachable node scores 0 rather than being omitted. Sharing one session is required anyway, because final-hop delivery needs a live session at the moment the packet arrives. + +**Alternative considered.** A single combined loop (`client -> GW -> agent-as-mix -> GW -> client`) exercising both directions per packet. Rejected because it halves the packet budget but destroys direction attribution, which is the entire reason for going minimal-hop. Also considered and rejected: separate assignments per phase, which would allow two agents to measure half a gateway each and produce results nobody can compose. + +**Consequence.** An agent that dies after phase 1 abandons the whole run rather than resuming it. Phase 1 involves no sphinx processing at the gateway at all (the client supplies an explicit next hop and the gateway forwards verbatim), so a phase-1 failure implicates the session, the bandwidth path, or the outbound forwarder, never the sphinx layer. Neither phase exercises `forward_hop_processing_enabled`, which is derived from `modes.mixnode` and is off on a pure gateway, so the tests can never accidentally depend on a gateway performing mix forwarding. + +### Decision 4: Gateway client sessions are established over `ws://` only + +**Choice.** The agent connects to `ws://{announced_ip}:{clients_ws_port}`, constructed directly from the refresher's stored data. Announced hostnames and wss entries are ignored. Testing the wss ingress is a future test kind. + +**Why.** Three properties fall out at once: the source IP the gateway observes is genuinely the agent's, so the monitor-session identification of Decision 5 is sound by construction rather than dependent on the operator's ingress topology; there is no DNS dependency, matching the minimal-hop design of every other probe; and there is no certificate handling. Authenticity is unaffected, because the registration handshake authenticates the gateway's ed25519 identity against the identity key the orchestrator holds from the mixnet contract. TLS would only protect a transport that the gateway protocol already protects with a handshake-derived shared key. + +**Alternative considered.** Reusing `NymNode::ws_entry_address()`. Rejected: it returns `wss://{hostname}` whenever the gateway announces one, and even its no-TLS variant prefers a hostname over an IP. Using it would reintroduce both DNS and TLS, and would put an operator-owned reverse proxy in the middle of a measurement whose scoring assumes the observed source IP is the agent's. + +**Confirmed since.** nym-node binds NO TLS listener at all: the clients websocket is started unconditionally in entry mode at `gateway_tasks.ws_bind_address`, and `announce_wss_port` is only an announcement field. So an announced wss entry always describes an externally terminated proxy, and the plain ws port is the node's own listener. `ws://ip` is therefore not merely the simpler target, it is the only way to measure the node rather than an operator's proxy. + +**Consequence.** This is a deliberate divergence from v1, which does probe the wss path when announced. A gateway with a broken TLS ingress therefore scores well under v3 liveness and badly under v1, systematically. Accepted, and made legible by bucketing the divergence gauge on whether the gateway announces a wss entry (Decision 12). In the entry model `clients_ws_port` is mandatory while `clients_wss_port` is optional, so plain ws is the base case and this leaves no coverage hole at the announcement level. + +### Decision 5: The node delivers monitor final-hop packets in-session or drops them, and treats a monitor client session as ephemeral and unmetered + +**Choice.** Two changes in nym-node, both driven by the already-propagated authorised-agent set, but identified differently because they sit on different protocols: + +1. A final-hop packet from an authorised monitor IP is processed rather than dropped, delivered to a live client session if there is one, and **never** written to disk storage. This stays source-IP keyed, like every other gate on the mixnet listener. +2. A client websocket session whose registration handshake authenticates an ed25519 identity announced on-chain for an authorised monitor is unmetered and persists nothing: no shared-keys row, no bandwidth row, no inbox. This is keyed on the verified identity, NOT on the source IP (Decision 14). + +**Why.** (1) is the enabling change for gateway egress testing, which is refused outright today. Suppressing the disk fallback is not an optimisation but a correctness and hygiene requirement: it gives the agent exactly the semantics it wants (a packet that did not arrive on the socket was not delivered, full stop), and it stops monitor traffic from accruing undeliverable rows on every gateway in the network at liveness cadence. (2) removes the need for the agent to hold ecash ticketbooks, which for a one-shot process would mean an external credential-provisioning pipeline for a probe that transfers a few kilobytes. + +**Alternative considered.** Real ticketbooks for agents, needing no node change. Rejected as disproportionate. Also considered: unmetered but still persisted, which is a smaller change but leaves a row per (agent, gateway) pair and keeps the inbox path live. + +**Confirmed since.** The fully non-persisting variant is reachable. `register_client` is the whole persistence surface of registration: it calls `insert_shared_keys` for the storage id, then `get_available_bandwidth` and `create_bandwidth_entry` with it. `BandwidthStorageManager` uses that id only inside storage calls, so with storage access disabled it is unused and can become optional. And the inbox push is keyed on the client ADDRESS rather than the id, so with monitor final-hop packets never stored it is a no-op query even if left in place; skipping it is hygiene rather than correctness. + +**Consequence.** An authorised monitor gains free unmetered gateway transit, and that transit is NOT confined to monitor destinations. The routing filter's "a monitor may only send to a monitor" rule applies to packets that arrive on the mixnet listener, where the monitor flag is set from the connection's source IP; a packet handed to the gateway over a client session is forwarded with that flag unset, so it routes anywhere an ordinary client's packet may go. The exemption is therefore an unconfined grant, which is precisely why it keys on a verified identity rather than on a source IP that k8s host ports share and CNI pools recycle (Decision 14). The `client_id` that the authenticated handler threads into `BandwidthStorageManager` is storage-assigned, so a non-persisting session needs that identity to become optional; this is the seam the change touches, not merely a skipped check. + +### Decision 6: An agent tests a wave of targets concurrently, and one wave is one concurrent batch + +**Choice.** An assignment hands the agent N targets which it establishes and probes concurrently. The lease stamped on the in-progress rows is `now + kind_budget + slack`, independent of N, because the agent runs the whole wave at once rather than in sub-waves. Aggregate send rate is configured as a total budget with the per-target rate derived from it, never the other way round. + +**Why.** Concurrency turns the lease bound from the sum of per-target worst cases into the maximum of them, which is what makes a full-network sweep at v1's cadence possible and what makes batch assignment safe at all. Sequentially, a 50-target batch of unresponsive nodes runs past any reasonable timeout, the orchestrator evicts the leases while the agent is still working, and the nodes get reassigned to a second agent, producing exactly the concurrent double-measurement the per-node mutex exists to prevent. Defining a wave as one concurrent batch keeps the orchestrator from having to know the agent's concurrency in order to size the lease. + +**Alternative considered.** One target per request in a loop until the assignment comes back empty. It needs no lease arithmetic at all and bounds a crash to one node, at the cost of one round trip per target (a low single-digit percentage of a multi-second test). Rejected only because concurrency raises sweep throughput by an order of magnitude, which is the binding requirement; the loop remains the fallback if wave concurrency proves troublesome. + +**Consequence.** The agent's tester is built around exactly one target today: one listener bound per run, a `NoiseNetworkView` holding one node's key, source-address acceptance gated on one node's addresses, and a packet processor built from one reusable header. A wave needs a shared listener, a Noise view holding every target's key, and per-target attribution. Attribution is by the source IP of the node's return connection, which is sound because node IPs are unique across the population; the agent's known-source set and Noise view for a wave are the union of every target's announced addresses, and that union is collision-free for the same reason. Measurements within a wave share the agent's NIC, CPU and scheduler, so latency figures from a wave are not comparable with stress-test latency figures (Decision 11). The existing "10k packets per second" figure was measured as batched writes down one connection and does not transfer to many sockets each carrying a low rate. Rather than block the work on measuring the fan-out shape, the aggregate budget is a configured knob carrying a provisional default, so an agent host that cannot sustain it is a configuration change rather than a code change. + +### Decision 7: Per-kind staleness and rotation, retained per-node mutex, leases materialised on the row + +**Choice.** A new `node_test_state (node_id, test_kind)` table holds `last_tested_at`, `last_testrun_id` and `last_tested_ip`, replacing `nym_node.last_testrun` and `nym_node.last_tested_ip`. `testrun_in_progress` keeps its `node_id` primary key and gains `expires_at` plus a `test_kind` column for observability. Eviction becomes `DELETE FROM testrun_in_progress WHERE expires_at < ?`. Liveness eligibility additionally requires that the node's stress-kind `last_tested_at` is older than a cooldown. + +**Why.** Per-kind state is what stops a 15-minute liveness cadence and a 2-hour stress cadence from fighting over one staleness pointer and one rotation cursor. Materialising the deadline on the row means the eviction sweep never needs to learn about kinds, so a future expensive kind that runs for minutes needs no eviction change; today's sweep takes a single cutoff derived from a global `test_timeout` and cannot express two budgets. Keeping the in-progress key on `node_id` alone gives the "one test at a time per node, across kinds" property for free. The cooldown covers the case the mutex cannot: a liveness test handed out the instant a stress test's row clears measures a node whose queues are still draining. + +**Alternative considered.** A per-kind in-progress table. Rejected: it would permit simultaneous stress and liveness measurement of one node, which biases both. + +**Consequence.** Denormalising `last_tested_at` also fixes an existing defect. Today staleness is read through `JOIN testrun tr ON tr.id = n.last_testrun` with `ON DELETE SET NULL`, so when eviction removes a node's last run the node reads as never-tested and jumps the queue. + +### Decision 8: Results carry per-signal rows under a run-level row + +**Choice.** `testrun` keeps run-level facts (kind, node, tested address, timing, error) and a new `testrun_signal (testrun_id, signal)` child table carries the counts and latency distributions per measured signal. A mixnode liveness or stress run has one signal; a gateway liveness run has two (`gw_ingress`, `gw_egress`). The score reported downstream is the average over the kind's fixed signal set. + +**Why.** Downstream consumers want one number per node, but an operator needs to tell "perfect ingress, dead egress" from "uniformly half-lossy", and averaging destroys that distinction. This is the same argument that put the tested address on each result during the dual-stack work: without it a per-address failure is indistinguishable from a dead node. A child table also stops `testrun` from growing a column group per future test kind. + +**Alternative considered.** Nullable per-direction columns on `testrun`. Simpler to query, worse to extend, and the proposal explicitly anticipates further test kinds. + +### Decision 9: Per-kind submission streams + +**Choice.** Each test kind has its own submission watermark and its own nym-api endpoint, and each endpoint keeps its own per-signer replay high-water mark. + +**Why.** The existing single `metadata.last_submitted_testrun_id` cannot serve two destinations; the first liveness submission would drag the stress watermark past unsubmitted rows. Separate per-signer high-water marks are equally load-bearing: nym-api rejects any batch whose timestamp is not strictly greater than the signer's last accepted one, so two interleaved streams signed by the same orchestrator identity and validated against one shared map would reject each other indefinitely. + +**Consequence.** The liveness ingest path repeats the staleness, membership, monotonicity and signature checks rather than sharing the stress path's state. + +### Decision 10: No further hardening of the IP-keyed monitor trust on the mixnet listener + +**Choice.** Monitor identification on the MIXNET listener, meaning the Noise responder gate, the routing filter and the replay bypass, stays keyed on the authorised agent's source IP. No same-IP forwarding restriction is added. The client-session exemption is the exception and is keyed on a verified identity instead (Decision 14). + +**Why.** An IP that is compromised can be de-authorised by the orchestrator, which bounds the exposure without new mechanism, and monitor-sourced packets on that listener are confined to monitor destinations by the existing routing filter. Moving the mixnet gates onto the Noise-authenticated x25519 static key is already an identified follow-on and remains the right place for that hardening. It cannot, however, be the answer for the client-session gate: `upgrade_noise_responder` is invoked only from the mixnet connection handler, so the client websocket port runs no Noise handshake and has no static key to authenticate. A gate placed there can never be fixed by the Noise follow-on, which is what forces Decision 14 rather than leaving it as future work. + +**Consequence.** Keeps `agent1 -> node -> agent2` possible, which is the only way to attribute a mixnode's loss to its inbound versus outbound direction, and which a same-IP restriction would have foreclosed. + +### Decision 11: Liveness scores delivery ratio only; latency is recorded, not scored + +**Choice.** The liveness score is the delivery ratio averaged over the kind's signals. The full RTT distribution keeps being recorded and submitted, but carries no weight. + +**Why.** Two confounds make a latency-derived score untrustworthy today. Nodes defer replay checking in batches bounded by `maximum_replay_detection_deferral` (50ms) and `maximum_replay_detection_pending_packets` (100), and only defer when the bloomfilter lock is contended, so a low-volume probe measures a busy node as slower than an idle one in a step function, penalising exactly the nodes that are carrying traffic. And measurements inside a wave include the agent's own queueing. Recording the distribution first means the weighting decision can be made against real data. + +**Consequence.** v1's absence of any latency signal is preserved for now, so the migration changes attribution without also changing what the score means. + +### Decision 12: Liveness ships as a third performance component at weight zero, with a divergence gauge + +**Choice.** Liveness enters node performance as a component alongside the v1 routing score and the v3 stress score, initially weighted zero, mirroring the existing `use_stress_testing_data` / `minimum_available_*` / `*_score_weight` gating trio. Alongside it, a gauge reports per-node divergence between v3 liveness and v1 routing, bucketed by whether the node announces a wss entry. + +**Why.** Two independent populations will score zero on liveness for reasons unrelated to their forwarding: nodes that have not ingested their agents' authorisations, and gateways not yet carrying the Decision 5 node changes. Shipping at weight zero makes both harmless, and the divergence gauge turns the eventual cutover into a measurement rather than a judgement call. Bucketing on the wss announcement separates the divergence this design knowingly introduces (Decision 4) from the divergence that indicates a real problem. + +**Consequence.** A liveness failure whose cause is a rejected Noise handshake still scores zero, because a node that will not accept a connection is not routable and "unmeasurable" must never score better than "measurably broken". During shadow mode the distinct failure outcomes are recorded so that the divergence population can be explained rather than merely counted. + +### Decision 13: The agent's client identity is derived from its noise key rather than provisioned + +**Choice.** The agent derives its ed25519 client identity deterministically from its existing x25519 noise private key via HKDF with a domain-separation label, using the output directly as the ed25519 seed. The random-per-test alternative is dropped, because Decision 14 requires the identity to be announced before it is used. + +**Why.** This adds no on-disk key material to provision and no operator step, which was the original point, and a stable identity is what makes the on-chain announcement possible at all. A labelled KDF rather than seeding a CSPRNG with the raw private key gives domain separation, so any future derivation from the same secret stays independent; this is key separation, which is what HKDF is for, not key reuse. As a side benefit, a stable identity keeps registration rows bounded at one per (agent, gateway) pair should the fully non-persisting session of Decision 5 turn out not to be reachable. + +**Consequence.** The noise key becomes a root secret, which is not an escalation because holding it already allows impersonating the agent to every node. Rotating the noise key rotates the client identity, which now also means re-announcing it. The agent's client identity is permanently recognisable to a gateway, which costs nothing: the agent's addresses are published on-chain, so liveness traffic is not covert regardless. + +### Decision 14: The agent's ed25519 client identity is announced on-chain, and the gateway session exemption keys on it + +**Choice.** The `network-monitors` contract is migrated to carry an OPTIONAL base58 ed25519 identity key on each agent entry, supplied by the orchestrator on `AuthoriseNetworkMonitor` and announced to the orchestrator by the agent alongside its address pair and noise key. A gateway grants the ephemeral unmetered session of Decision 5 only to a client whose registration handshake authenticates an identity present in that on-chain set. The source IP plays no part in this gate. + +**Why.** The gateway registration handshake is already mutually authenticating: the gateway takes the client's ed25519 identity from the initialisation message and, at step 5, decrypts and verifies the client's signature over both ephemeral DH public keys against it. By the time a session exists the gateway therefore holds a possession-proven client identity, so announcing that identity turns the exemption into a set-membership test on a key the gateway has already verified. No new protocol message, no challenge to design, no client protocol version negotiation. Against that, the alternative gate is a source IP that several agents share through distinct k8s host ports and that CNI pools recycle, guarding an exemption which grants unconfined unmetered mixnet transit (Decision 5). The gate that most needs a real identity is the one where an identity is already sitting unused. + +The migration is fleet-safe in the exact sense the Context constraint requires. An additive field on the existing `AuthoriseNetworkMonitor` variant and on the stored entry is silently ignored by un-upgraded nodes, unlike a new variant. Making it `Option` means no data migration and no backfill logic: the contract's agent save is an upsert and agents re-announce before every run, so every live agent's entry acquires its identity within one liveness cycle. And the consumer that needs the field is a gateway that must be upgraded for Decision 5 regardless, so the change adds a governance action rather than a second fleet rollout, which is what distinguishes this from the earlier decision to defer contract work. + +**Alternative considered.** An orchestrator-signed capability token presented at registration, verified against the orchestrator identity key that the contract already stores, needing no contract change at all. Genuinely attractive on revocation, where a short-lived scoped token beats an on-chain entry that only a multisig transaction can remove. Rejected on total cost: it needs a slot in the versioned client control protocol and a new orchestrator-identity cache on every node, where the on-chain identity needs neither, because the handshake already carries and proves the key. Worth revisiting if the client protocol is being changed for other reasons. + +**Alternative considered.** Going further and REMOVING agent IP addresses from the contract, so the monitor fleet is not publicly enumerable. The Noise pattern permits it: `Noise_XKpsk3` transmits the initiator's static key in message 3 with `se` proving possession, and the PSK is derived from the responder's own public key, so a node needs no advance knowledge of an initiator to complete a handshake and learn its identity. Rejected for three reasons. The routing filter's destination set is `HashSet` and the mixnode probe's return hop is a fresh outbound connection, so a node that does not know the agent's address cannot send the probe back at all; replacing the set with a per-connection "may return to this peer" capability would work but forecloses the `agent1 -> node -> agent2` shape Decision 10 preserves. Dropping the address also means rekeying agent storage and retyping `RevokeNetworkMonitor`, which is the fleet-unsafe kind of change: an un-upgraded node would fail to parse revocations and keep a stale bypass indefinitely. And the benefit is convenience rather than secrecy, since any operator can read the monitor set out of their own node, and the retained `reuse_header` replay bypass forces the node to classify a connection as a monitor at packet time anyway. Real indistinguishability needs fresh headers, real ticketbooks and no special treatment, which is a different and much larger design. + +**Alternative considered.** Since the message is being touched anyway, consolidating the agent's address pair into ONE `AuthoriseNetworkMonitor` carrying explicit ipv4 and ipv6 fields, instead of the two messages sent in one transaction today. Rejected, because the two-message shape is what carries dual-stack authorisation to un-upgraded nodes rather than a workaround for the frozen contract. A node dispatches `handle_msg` per message within a transaction, so an un-upgraded node already applies both authorisations independently without knowing that dual-stack exists. Renaming `mixnet_address` into a v4/v6 pair breaks deserialisation on those nodes, which log and continue and therefore silently stop learning about every agent; keeping `mixnet_address` and adding an optional v6 field parses, but an un-upgraded node then authorises the ipv4 address only, silently reintroducing the ipv6 rejection this subsystem already fixed. The consolidation also buys less than it appears: atomicity comes from the transaction, not from the message count, so the only saving is the `execute_multiple` call site. + +The version that WOULD simplify meaningfully is one entry per agent keyed by the noise key and holding both addresses, which deletes the noise-key group-by in the orchestrator's cache rehydration, makes noise-key uniqueness enforceable, and turns an address change into an update instead of an insert-plus-orphan. That is a storage rekey plus a retyped `RevokeNetworkMonitor` plus a changed paged-query response, and since a failed startup load aborts node startup, an un-upgraded node would fail to start after such a migration. It therefore has to be staged across releases in the manner the contract capability's schema-evolution requirement describes, and it is a separate change that liveness does not need. + +**Consequence.** The exemption survives a change of the agent's egress address, which matters because pods are rescheduled onto recycled addresses and the orchestrator still has no revocation path. A gateway that has not ingested an agent's identity meters that agent's session and the run scores zero on both phases, which is the same failure mode as a node that has not ingested an authorisation and is already covered by shipping at weight zero. The node gains a third derived structure alongside the routing set and the noise map, keyed by identity rather than by IP, populated from the same startup load and the same websocket events. Nothing else in the change consumes the field: the mixnet gates keep their IP keying (Decision 10), and assignment, scheduling and submission are unaffected. + +## Risks / Trade-offs + +- **A node that has not ingested its agent authorisations scores zero on liveness.** This is the migration's central risk, because unlike v1 the probe requires the tested node to have explicitly allowlisted the prober. → Ship at weight zero with a divergence gauge (Decision 12), and treat the already-identified node-side periodic reconciliation follow-on as a prerequisite for the cutover rather than as optional hardening. +- **Liveness traffic is not covert at the mixnode, where v1's was.** A dishonest node can forward monitor traffic perfectly and drop everything else, because the authorised agent set is public on-chain. → Accepted deliberately: the current risk is negligible and the route-conflation bias being fixed is real and observed. The mitigation, if it is ever needed, is many agents across many providers, or an occasional multi-hop client-path run reusing the gateway-client capability this change introduces. +- **Un-upgraded gateways score zero on the egress phase**, since the final-hop drop is the current behaviour. → Weight zero until the fleet has upgraded; the divergence gauge measures when that is true. +- **A gateway with a broken wss ingress scores well on liveness and badly under v1.** → Known and intended (Decision 4); made legible by the wss bucket in the divergence gauge, and closed later by a dedicated wss test kind. +- **A gateway whose ws listener is not dual-stack scores zero on its ipv6 rotation.** → Reported as a genuine finding rather than suppressed, since it means the gateway does not serve ipv6 clients, but it will present as a population of half-scoring gateways on day one and should be expected. +- **Free unmetered gateway transit for an authorised monitor, unconfined by the routing filter**, since a packet entering through a client session does not carry the monitor flag that restricts monitor traffic to monitor destinations. → Keyed on a handshake-verified on-chain ed25519 identity rather than on a shared or recycled source IP (Decision 14), and bounded by revocation. +- **A gateway that has not ingested an agent's announced identity meters that agent's session**, so the run scores zero on both phases rather than one. → Same failure mode and same mitigation as a missed authorisation event: weight zero plus the divergence gauge, with the periodic-reconciliation follow-on as a cutover prerequisite. +- **Wave concurrency shares the agent's link across targets**, so one saturated target's back-pressure and the agent's own scheduling contaminate its neighbours' latency figures. → Aggregate rate budget rather than per-target times width, and latency excluded from scoring (Decision 11). +- **A crashed agent locks a whole wave** until its lease expires. → Leases are per-kind and sized to a single concurrent wave, so the exposure is tens of seconds rather than the current global five-minute timeout; results submitted per target release their own locks as the wave progresses. +- **The gateway session exemption assumes no proxy in front of the plain ws port.** → Made structural by connecting to `ws://ip` (Decision 4); an out-of-bandwidth failure must be recorded as a distinguishable outcome so a proxied gateway is diagnosable rather than silently scored zero. + +## Migration Plan + +1. Migrate the contract to carry the optional agent ed25519 identity, and land the orchestrator and agent sides of the announcement so that live agents start populating the field through the existing upsert. This is inert for every consumer until step 3 reaches a gateway, and safe for un-upgraded nodes because the field is additive. +2. Land the orchestrator schema migration and per-kind scheduling with liveness assignment disabled, so the stress test keeps running on the new tables. The migration moves `nym_node.last_testrun` and `last_tested_ip` into `node_test_state` under the stress kind, and backfills `expires_at` on any live in-progress rows. +3. Land the nym-node changes (final-hop delivery policy, ephemeral unmetered monitor session keyed on the announced identity) and let them propagate through the fleet. They are inert until an agent exercises them. +4. Land the agent's liveness profile and wave concurrency, and enable mixnode liveness. This needs no gateway-side change and validates the wave machinery on the larger population. +5. Enable gateway liveness once enough of the fleet carries step 3. +6. Land nym-api ingest and the shadow-weighted component at weight zero, with the divergence gauge. +7. Decide the weighting, and separately the v1 cutover, from the divergence data. Both are later changes. + +Rollback at any step is a config change rather than a revert: liveness assignment can be disabled in the orchestrator, and the component weight is already zero. The contract migration is not reversible, but it is inert on its own: an announced identity that no gateway consults changes nothing. The nym-node changes are the only step that is not trivially reversible in a running fleet, which is why they carry no behaviour for anyone but authorised monitors. + +## Resolved Questions + +1. **Liveness packet count and per-target rate.** RESOLVED by decision rather than by measurement: the profile ships with provisional defaults chosen for score granularity, and every value is a configured knob. A per-target count of 100 gives 1% granularity and roughly 2.2% binomial noise at a true 95% delivery, against v1's three packets per route at 33% granularity, so it is a thirtyfold increase in evidence per node and a sensible floor. The aggregate rate budget and wave size start at 500 packets/second and 20 targets. Deliberately NOT measured first: the agent hosts are not the machines these numbers would be measured on, so an agent that cannot sustain the budget is a configuration change. The sweep arithmetic that the knobs must satisfy is `T_send = count x wave / aggregate_rate` and `population / wave <= invocations x (interval / T_wave)`. +2. **May a single agent invocation mix kinds?** RESOLVED, no, by construction: an invocation takes exactly ONE assignment, which is either a single stress target or a single liveness wave, then exits. No stickiness rule is needed. The residual concern is different from the one originally recorded: consecutive invocations on the same HOST, where a liveness wave measures an agent still recovering from a stress test and charges the loss to every node in the wave. Not addressed now, because 30000 packets at 1000pps is around 16 Mbps for a 2KB packet and saturates nothing on a container host, and sockets die with the process. If deployment ever shows contaminated waves, the remedy is a cooldown keyed on the agent's IP (not its socket address, since several agents share a host NIC behind distinct ports), symmetrical to the per-node `liveness_after_stress_cooldown`. +3. **Does a wss-configured gateway still bind its plain ws port?** RESOLVED, yes, and more strongly than the question assumed: nym-node binds no TLS listener at all, so an announced wss entry always denotes an externally terminated proxy. See Decision 4. +4. **How ephemeral can the monitor session actually be?** RESOLVED, fully non-persisting is reachable, with the storage-assigned client id becoming optional. See Decision 5. +5. **Does the gateway's session path have access to the derived monitor set?** RESOLVED, it needs its own handle rather than a read of the mixnet structures, because the gateway client handling lives in a different crate from the routing set and noise map. `CommonHandlerState` already carries live shared handles passed down from nym-node (`upgrade_mode`, `active_clients_store`), so the identity set becomes one more field on it plus a builder setter, populated from the same startup load and websocket events. diff --git a/openspec/changes/network-monitor-liveness-tests/proposal.md b/openspec/changes/network-monitor-liveness-tests/proposal.md new file mode 100644 index 00000000000..f1d39c16277 --- /dev/null +++ b/openspec/changes/network-monitor-liveness-tests/proposal.md @@ -0,0 +1,39 @@ +## Why + +The nym-api-internal v1 network monitor scores a node's "liveness" by looping packets through a route of five nodes (entry gateway, three mixnodes, exit gateway) and attributing the whole route's delivery ratio to whichever node was substituted in. Its own design record predicted the consequence we now observe in production: "a node that is fine but happens to be tested only against a marginal route can score low", and with `minimum_test_routes = 1` a score "can rest on a single route/gateway path and be noisy". Each node also receives only `per_node_test_packets` (3) packets per route, so a mixnode's reliability is quantised to ninths. + +Network monitor v3 already has the machinery to fix this: chain-authorised agents that probe a single node directly and attribute the result to that node alone. Its design record reserves exactly this work, naming "single-hop LIVENESS checks for mixnodes AND gateways, in order to REPLACE the nym-api-internal v1 network monitor" as the known roadmap, and the current implementation already carries the seams (a gateway test type in the data model, gateway-capable nodes retained in the registry, `is_mixnode` / `was_reachable` on every result, and a node-type-agnostic authorisation model). This change builds the first half of that: liveness as a second test kind alongside stress testing, scored per node with no other node in the path. + +## What Changes + +- **A new `liveness` test kind** alongside the existing stress test, assigned by the orchestrator and executed by the agent. For a mixnode it is the existing two-hop self-loop probe at low volume (order 100 packets rather than 30000). For a gateway it is a new two-phase test. +- **Gateway liveness is one indivisible test with two phases** over a single client session: an ingress phase (agent as gateway client, packet forwarded out to the agent acting as a mixnode) and an egress phase (agent as mixnode, packet delivered as a final hop back to the agent's client session). Both phases MUST be performed by the same agent in the same run; a phase that produces no signal scores zero rather than being excluded from the average. +- **Gateway client sessions are established over `ws://:` only**, ignoring announced hostnames and wss entries. This keeps the tested source IP truthful, removes DNS and certificate handling from the probe, and costs nothing in authenticity because the registration handshake already authenticates the gateway's ed25519 identity. Testing the wss ingress path is explicitly out of scope and recorded as a future test kind. +- **BREAKING (nym-node):** final-hop packets originating from an authorised network-monitor agent are currently dropped outright ("unsupported network monitor final hop packets"). They MUST instead be processed and delivered to a live client session, and MUST NOT fall back to disk storage. Gateway egress testing is impossible without this, and the no-disk rule keeps monitor traffic from accruing undeliverable rows on every gateway. +- **BREAKING (nym-node):** a client websocket session whose registration handshake authenticates an ed25519 identity announced on-chain for an authorised network-monitor agent MUST be treated as an ephemeral monitor session: unmetered, and writing nothing to gateway storage. The agent presents no ecash ticketbook. +- **A contract migration adds an optional ed25519 identity key to each agent entry**, announced by the agent and written by the orchestrator in the existing authorisation transaction. This is what lets the session exemption above key on a cryptographically verified identity rather than on a source IP that k8s host ports share and CNI pools recycle, and it is the gate where an identity is already available: the gateway registration handshake proves possession of the client's ed25519 key before any session exists. The field is additive and optional, which un-upgraded nodes ignore, and it needs no data migration because the contract's agent save is an upsert and agents re-announce before every run. +- **The agent tests a wave of targets concurrently** rather than one target per invocation, turning the assignment lease bound from the sum of per-target worst cases into the maximum of them, and making a full-network liveness sweep viable at v1's cadence. This requires a shared ingress listener, a multi-target Noise view, and per-target attribution of returned packets. +- **Per-test-kind scheduling in the orchestrator**: per-kind staleness gates, per-kind address rotation cursors, per-kind lease budgets materialised as an `expires_at` on each in-progress row, and a cooldown that keeps a liveness test from measuring a node still recovering from a stress test. The existing single-in-flight-test-per-node mutex is retained unchanged and now spans kinds. +- **Per-kind result submission** to nym-api, with a separate watermark per kind and a per-signer replay high-water mark that cannot be shared between kinds. +- **Liveness enters node performance as a third component with weight zero** (shadow mode) alongside the v1 routing score and the v3 stress score, plus a divergence gauge comparing v3 liveness against v1 routing, bucketed by whether the gateway announces a wss entry so that expected divergence is separable from unexpected divergence. + +## Capabilities + +### New Capabilities + +None. Liveness testing is performed by the same actors, under the same chain authorisation, through the same assignment and submission lifecycle as stress testing, so it belongs in the existing capability rather than duplicating that context. + +### Modified Capabilities + +- `network-monitors-contract`: adds an optional base58 ed25519 identity key to the agent entry and to `AuthoriseNetworkMonitor`, validated on shape and populated through the existing upsert rather than by a data migration; and records the schema-evolution rule that keeps un-upgraded nodes working, namely additive optional fields only, never a new or retyped `ExecuteMsg` variant. +- `nym-network-monitor`: adds the liveness test kind and its per-node and per-gateway probe mechanics; amends testrun assignment to be keyed by (node, test kind) with per-kind staleness, rotation and lease budgets; amends the agent lifecycle from one target per invocation to a concurrently-executed wave; amends the node-side gating requirement to permit monitor final-hop delivery without disk fallback and to define the ephemeral unmetered monitor client session; amends the orchestrator storage requirement for the per-kind schema; amends result submission for per-kind watermarks; amends the downstream consumer surface for the shadow-weighted liveness component; and amends the configuration surface for the liveness knobs. + +## Impact + +- **nym-node** (`src/node/mixnet/handler.rs`): final-hop handling for network-monitor packets. **BREAKING** for gateway egress testing, which cannot work on un-upgraded nodes. Un-upgraded gateways will score zero on the egress phase, which is one reason liveness ships at weight zero. +- **gateway** (`src/node/client_handling/websocket/connection_handler/{fresh,authenticated}.rs`) and **common/credential-verification** (`bandwidth_storage_manager.rs`): ephemeral, unmetered monitor client sessions. Touches the `ClientDetails` / `BandwidthStorageManager` seam because the storage-assigned `client_id` has no meaning for a session that persists nothing. Also needs a third derived structure alongside the existing routing set and noise map: the announced monitor identities, keyed by ed25519 key rather than by IP, populated from the same startup load and the same nyxd websocket events. +- **nym-network-monitor-orchestrator**: new migration (per-kind test state, per-kind watermarks, in-progress leases, gateway ws entry address, per-signal result rows), assignment query, node refresher, result submitter, prometheus surface. +- **nym-network-monitor-agent**: liveness probe profile, wave concurrency with a shared listener and multi-target Noise view, gateway client session and the two-phase gateway probe, ed25519 client identity (derived, not provisioned). +- **nym-network-monitor-orchestrator-requests**: test-kind-tagged assignments and results carrying per-signal breakdowns, and the announced ed25519 identity on the announce request. +- **nym-api**: liveness ingest endpoint with its own per-signer replay high-water mark, storage for liveness results, and a shadow-weighted performance component plus the divergence gauge. +- **contracts/network-monitors** and **common/cosmwasm-smart-contracts/network-monitors-contract**: an optional `bs58_ed25519_identity` on `AuthoriseNetworkMonitor` and on the stored `AuthorisedNetworkMonitor`, validated as base58 decoding to 32 bytes. The contract cannot be redeployed but it can be migrated: it carries a cw2-guarded `migrate` entry point and its admin is the Nymtech SA multisig, so this is a governance action plus a schema bump with no data migration. The field is deliberately additive and optional, because `cw_serde` does not set `deny_unknown_fields` and an un-upgraded node therefore ignores an unknown field on a variant it knows, whereas a new or retyped `ExecuteMsg` variant would fail to parse and make that node silently stop learning about agents. Requires a multisig migration proposal, and no second fleet rollout: the only consumer of the field is a gateway that must already be upgraded for the changes above. diff --git a/openspec/changes/network-monitor-liveness-tests/specs/network-monitors-contract/spec.md b/openspec/changes/network-monitor-liveness-tests/specs/network-monitors-contract/spec.md new file mode 100644 index 00000000000..392edc35813 --- /dev/null +++ b/openspec/changes/network-monitor-liveness-tests/specs/network-monitors-contract/spec.md @@ -0,0 +1,71 @@ +## ADDED Requirements + +### Requirement: Schema evolution MUST remain parseable by un-upgraded consumers + +The contract's message and response types SHALL only ever be extended in ways that an un-upgraded third-party consumer can still parse. Concretely: a new field on an existing `ExecuteMsg` variant, or on a stored type carried in a query response, is PERMITTED and MUST be optional; introducing a NEW `ExecuteMsg` variant, or changing the type of an existing field, MUST be treated as a breaking fleet change and MUST NOT be used to deliver behaviour that un-upgraded nodes are required to keep observing. + +The reason is asymmetric failure. Contract types use `cw_serde`, which does NOT set `deny_unknown_fields`, so a consumer compiled against an older schema silently ignores an unknown field on a variant it recognises and continues to apply the message. A consumer that meets an unrecognised variant, or a field whose type no longer matches, fails deserialisation instead; a Nym node's contract event handler treats that failure as non-fatal, logs that the schema may have changed, and continues processing later blocks. The observable result is not a loud error but a node that has silently stopped learning about agent authorisations and revocations, keeping a stale replay bypass for a revoked address indefinitely. + +A change that genuinely requires a new or retyped variant MUST therefore be staged: add the new form alongside the old, wait for the node fleet to carry it, and only then retire the old form. + +#### Scenario: An added optional field does not disturb an un-upgraded node +- **WHEN** an orchestrator sends `AuthoriseNetworkMonitor` carrying a field that a node's compiled schema does not know +- **THEN** that node parses the message, ignores the unknown field, and still authorises the agent + +#### Scenario: A new variant would silently strand an un-upgraded node +- **WHEN** a hypothetical new `ExecuteMsg` variant is used to authorise or revoke an agent +- **THEN** an un-upgraded node fails to deserialise it, logs that the schema may have changed, continues with later blocks, and never applies the authorisation or revocation + +## MODIFIED Requirements + +### Requirement: Only an authorised orchestrator may authorise agents, keyed by socket address as an upsert + +`AuthoriseNetworkMonitor { mixnet_address, bs58_x25519_noise, noise_version, bs58_ed25519_identity }` MUST be orchestrator-only, failing with `NotAnOrchestrator` for any other sender. `bs58_x25519_noise` MUST be validated as base58 decoding to exactly 32 bytes (an x25519 noise key), failing with `MalformedX25519AgentNoiseKey` otherwise. On success it MUST save an `AuthorisedNetworkMonitor` keyed by `mixnet_address`, recording `authorised_by = info.sender`, `authorised_at = env.block.time`, and the supplied noise key and version. The save MUST be an upsert: re-authorising the same socket address renews the entry (including `authorised_at`), in contrast to orchestrator authorisation which is a no-op for an existing entry. + +`bs58_ed25519_identity` MUST be OPTIONAL, and when present MUST be validated as base58 decoding to exactly 32 bytes (an ed25519 public key), failing with a dedicated malformed-identity error otherwise. It records the ed25519 client identity the agent presents when it opens a gateway client session, which is what allows a gateway to grant an unmetered monitor session against a cryptographically verified identity instead of a source IP. The contract MUST NOT require it, MUST NOT infer it, and MUST NOT treat its absence as an error: an entry without one is a validly authorised agent that simply cannot be recognised on the client-session path. Because the save is an upsert and agents re-announce before every test run, entries written before the field existed acquire it without any data migration or backfill. + +The contract places NO uniqueness constraint on `bs58_x25519_noise` OR on `bs58_ed25519_identity`: the same noise key MAY appear under several socket addresses, and does so by design, because a single agent authorises one ipv4 and one ipv6 address so that nodes accept its probes over either family. The registry therefore holds roughly TWO entries per agent, both carrying that agent's noise key and, once announced, the same identity key, and nothing on-chain records that a pair of entries belongs to one agent. + +An off-chain consumer that needs to recover which entries belong to one agent MUST group them by that noise key; the two entries of one agent are NOT adjacent in the pagination order, which sorts ipv4 before ipv6. The nym-network-monitor orchestrator does exactly this when it rehydrates its agent cache after a restart. That grouping is only sound as long as distinct agents never share a noise key, and the contract does not enforce it, so this is an assumption held by the consumer rather than an on-chain guarantee. A consumer that builds a set of authorised monitor identities MUST likewise tolerate the same identity arriving from several entries. + +#### Scenario: One agent's two addresses are two independent entries +- **WHEN** an orchestrator authorises one ipv4 and one ipv6 address for the same agent, both with its noise key +- **THEN** the registry holds two entries sharing that noise key, each independently revocable, with nothing on-chain marking them as one agent + +#### Scenario: Only orchestrators can authorise agents +- **WHEN** an account that is not an authorised orchestrator sends `AuthoriseNetworkMonitor` +- **THEN** the call fails with `NotAnOrchestrator` + +#### Scenario: A malformed noise key is rejected on shape +- **WHEN** the supplied `bs58_x25519_noise` is not valid base58 or does not decode to exactly 32 bytes +- **THEN** the call fails with `MalformedX25519AgentNoiseKey` + +#### Scenario: Re-authorising the same agent renews the entry +- **WHEN** an orchestrator authorises an agent for a socket address that already has an entry +- **THEN** the entry is overwritten with the new `authorised_by`, `authorised_at`, noise key, version, and identity key + +#### Scenario: An omitted identity key is accepted +- **WHEN** an orchestrator authorises an agent without supplying `bs58_ed25519_identity` +- **THEN** the entry is saved with no identity recorded, and the agent is authorised for every gate that does not depend on one + +#### Scenario: A malformed identity key is rejected on shape +- **WHEN** the supplied `bs58_ed25519_identity` is not valid base58 or does not decode to exactly 32 bytes +- **THEN** the call fails with a malformed-identity error and nothing is saved + +#### Scenario: An entry predating the field acquires it on the next announcement +- **WHEN** an agent whose stored entry has no identity re-announces and is authorised again +- **THEN** the upsert records its identity, with no data migration involved + +### Requirement: Migration refreshes build information only + +`MigrateMsg` MUST be an empty message. `migrate` MUST refresh build information and MUST guard against a downgrade or a wrong contract name via cw2 (`ensure_from_older_version`), and MUST perform no data migration. The `queued_migrations` module MUST contain no migration logic. + +This MUST remain true across the addition of the optional agent identity key. That field is deliberately shaped so that no stored entry needs rewriting: an absent value deserialises as `None` under the new schema, and the existing upsert populates it as agents re-announce. A future contract change that cannot be expressed this way MUST add its logic to `queued_migrations` rather than relaxing this requirement silently. + +#### Scenario: Migration performs no data rewrite +- **WHEN** the contract is migrated to a version carrying the optional agent identity key +- **THEN** build information is refreshed, the cw2 version guard runs, and no agent entry is read or rewritten + +#### Scenario: Pre-existing entries remain readable after the migration +- **WHEN** an agent entry stored before the migration is queried afterwards +- **THEN** it deserialises with no identity key and every other field unchanged diff --git a/openspec/changes/network-monitor-liveness-tests/specs/nym-network-monitor/spec.md b/openspec/changes/network-monitor-liveness-tests/specs/nym-network-monitor/spec.md new file mode 100644 index 00000000000..f259f8d4d84 --- /dev/null +++ b/openspec/changes/network-monitor-liveness-tests/specs/nym-network-monitor/spec.md @@ -0,0 +1,511 @@ +## ADDED Requirements + +### Requirement: Liveness is a second test kind whose mixnode probe is the stress probe at a low-volume profile + +The subsystem SHALL support a second test kind, `liveness`, alongside `stress`. Every unit of work MUST be identified by a `test_kind`, and the orchestrator MUST be the party that decides which kind a given assignment carries; an agent MUST support every kind. + +For a mixnode (or a `mixnode_and_gateway` node), a liveness probe SHALL be the same two-hop self-loop probe as a stress test - route `[tested_node, this_agent]`, `AckPacket`-sized mix packets, the same connectivity and bloomfilter probe sequence, the same `reuse_header` behaviour - executed under its own low-volume profile (`liveness_packets`, `liveness_target_rate`, `liveness_waiting_duration`) rather than the stress profile. The reported sent count MUST be forced to the profile's expected packet count on success, exactly as for a stress test, so a node that applies back-pressure to a liveness probe is penalised rather than flattered. + +Because the probe traverses exactly one node besides the agent, a liveness score MUST depend only on that node and the agent's own link. No other node of the network may appear in the measured path. + +#### Scenario: A mixnode liveness probe measures only the tested node +- **WHEN** a liveness assignment for a mixnode is executed +- **THEN** the packets traverse only the tested node and return to the agent, so no third node's behaviour can affect the score + +#### Scenario: The liveness profile is used rather than the stress profile +- **WHEN** a liveness probe runs against a mixnode +- **THEN** it sends `liveness_packets` packets at `liveness_target_rate` and waits `liveness_waiting_duration` for stragglers, leaving the stress profile's values untouched + +#### Scenario: Back-pressure on a liveness probe is penalised +- **WHEN** a node throttles the agent so that fewer than the expected packets are pushed, and the probe otherwise completes +- **THEN** the reported sent count is the expected count, lowering the node's delivery ratio + +### Requirement: Gateway liveness is one indivisible test of two phases over a single client session + +A gateway liveness assignment SHALL be a single, indivisible unit of work performed by ONE agent within ONE gateway client session, comprising two phases: + +1. **Ingress** - the agent, acting as a gateway client, submits a `ForwardSphinx` request whose next hop is the agent's own mixnet address, and counts the packets that arrive at its mixnet listener. This exercises the client session, the bandwidth path, and the gateway's outbound forwarder (including its Noise handshake as initiator). The gateway performs NO sphinx processing on this path: the client supplies an explicit next hop and the gateway forwards the packet verbatim, so an ingress failure implicates the session or the forwarder and never the sphinx layer. +2. **Egress** - the agent, acting as a mixnode, sends packets to the gateway's mixnet listener as FINAL-hop packets addressed to its own client session, and counts the packets pushed back to it over that session. This exercises the gateway's mixnet ingress, its final-hop sphinx unwrapping, its destination resolution, and its delivery to a live client. + +The client session MUST be established before either phase and held open through the drain window of both, because final-hop delivery requires a live session at the instant the packet arrives. The two phases MUST NOT be assignable, executable, or submittable separately. + +A gateway liveness run MUST produce a signal for each phase. The score denominator MUST be fixed by the test kind at two signals, so a phase that produces no signal scores zero rather than being excluded from the average. A phase-1 failure MUST NOT abort the run; only failure to establish the client session aborts it, in which case both signals are zero. Neither phase relies on `forward_hop_processing_enabled`, which is derived from a node's mixnode mode and is disabled on a gateway-only node. + +#### Scenario: Both phases run in one session by one agent +- **WHEN** a gateway liveness assignment is handed out +- **THEN** a single agent establishes one client session and performs both the ingress and egress phases within it, and no other agent is ever assigned either phase of that run + +#### Scenario: A dead egress path is not hidden by a healthy ingress path +- **WHEN** the ingress phase returns every packet and the egress phase returns none +- **THEN** the run records one full and one zero signal, and the reported score is their average over the kind's fixed two-signal denominator + +#### Scenario: An ingress failure still yields an egress measurement +- **WHEN** the gateway fails to forward the ingress phase's packets +- **THEN** the run continues into the egress phase and records the ingress signal as zero, rather than aborting + +#### Scenario: A session that cannot be established scores zero on both phases +- **WHEN** the agent cannot establish its client session with the gateway +- **THEN** the run aborts with the failure recorded and both signals are zero + +### Requirement: Gateway client sessions are established over plain ws at an announced ip address + +The agent SHALL establish its gateway client session at `ws://:`, constructed from the address the orchestrator assigned and the gateway's announced client websocket port. It MUST ignore any announced hostname and any announced wss entry, and MUST NOT reuse a topology helper that prefers either. + +This is required so that the session reaches the node itself rather than an operator-owned TLS terminator or reverse proxy, and so that the probe carries no DNS or certificate dependency. A Nym node binds NO TLS listener: its clients websocket is bound unconditionally in entry mode and the announced wss port is an announcement field only, so an announced wss entry always denotes externally terminated TLS and the plain ws port is the node's own listener. Authenticity is unaffected: the registration handshake authenticates the gateway's ed25519 identity, which the orchestrator holds from the mixnet contract, so no transport-level certificate is needed to know which gateway answered. + +A gateway liveness run MUST use ONE announced address for both of its phases, so that a run exercises a single address family end to end, consistent with the per-address rotation. Testing a gateway's wss ingress path is explicitly NOT covered by this kind and is recorded as intended future work (a separate test kind), which means a gateway whose plain-ws path works while its wss path is broken WILL score well here. + +#### Scenario: The session targets an ip, not a hostname +- **WHEN** a gateway announces both a hostname with a wss port and a plain client websocket port +- **THEN** the agent connects to `ws://:` and neither resolves the hostname nor negotiates TLS + +#### Scenario: The gateway is still authenticated +- **WHEN** the session is established without TLS +- **THEN** the gateway's identity is verified through the registration handshake against the identity key recorded for that node + +#### Scenario: Both phases use one address family +- **WHEN** the rotation selects a gateway's ipv6 address for a run +- **THEN** both the client session and the mixnet leg use that ipv6 address, and the sphinx return hop carries the agent's ipv6 address + +### Requirement: A liveness assignment is a wave of targets probed concurrently + +A liveness assignment SHALL carry a wave of targets that the agent probes CONCURRENTLY, and one wave MUST be one concurrent batch: the agent MUST NOT split a wave into sequentially-executed sub-waves. A stress assignment remains a single target. + +Concurrency is required so that the assignment's lease is bounded by the WORST CASE OF ONE target rather than by the sum over the wave, which is what makes a full-population sweep feasible at liveness cadence and what makes a multi-target lease safe at all. + +To execute a wave the agent MUST bind ONE shared ingress listener, MUST build a Noise view containing EVERY target's noise key keyed by every address that target is known by, and MUST treat the union of every target's announced addresses as its known-source set. Returned packets MUST be attributed to a target by the source address of the connection they arrive on, which is sound because node ip addresses are unique across the node population (unlike agent addresses, which may share an ip and be disambiguated only by port). The union is collision-free for the same reason. + +The send rate MUST be configured as an AGGREGATE budget across the wave with the per-target rate derived from it, never as a per-target rate multiplied by the wave width, so that widening a wave cannot silently multiply the agent's egress load. + +Every result MUST be submitted as soon as its own target completes, rather than at the end of the wave, so that each target's in-flight lock is released independently and an agent that dies mid-wave loses only the targets it had not yet reported. + +#### Scenario: A wave's duration is bounded by its slowest target, not their sum +- **WHEN** a wave contains several unresponsive targets +- **THEN** they time out concurrently, so the wave completes within roughly one target's worst case rather than the sum of them + +#### Scenario: Returned packets are attributed to the right target +- **WHEN** several targets in one wave return packets to the shared listener at the same time +- **THEN** each packet is attributed to the target whose address the connection originated from + +#### Scenario: Aggregate load does not scale with wave width +- **WHEN** the wave width is increased +- **THEN** the per-target rate falls so that the aggregate send rate stays within its configured budget + +#### Scenario: A crashed agent only holds its unreported targets +- **WHEN** an agent completes part of a wave and then dies +- **THEN** the completed targets have already been submitted and unlocked, and only the unreported ones wait for the lease to expire + +### Requirement: The agent's gateway client identity is derived, announced on-chain, and verified by the gateway + +The agent SHALL obtain the ed25519 client identity it needs for a gateway client session WITHOUT any additional on-disk key material, by deriving it deterministically from its existing x25519 noise private key using a labelled key-derivation function whose output is used directly as the ed25519 seed. The derivation MUST carry a domain-separation label so that it can never collide with any other value derived from the same secret. Seeding a general-purpose random number generator with the raw private key bytes MUST NOT be used, because it provides no such separation. + +The identity MUST be STABLE rather than freshly generated per test, because it is announced before it is used: the agent SHALL include its base58 ed25519 identity public key in its announcement to the orchestrator, and the orchestrator SHALL record it on-chain with the agent's authorisation. A gateway SHALL grant the ephemeral unmetered monitor session ONLY to a client whose registration handshake authenticates an identity present in the on-chain authorised-agent set, and MUST NOT grant it on the basis of the connection's source IP. + +Keying this gate on the identity rather than the source IP is required, not preferred. The registration handshake already possession-authenticates the client's ed25519 key before any session exists, so the check costs no new protocol; the exemption it guards grants unmetered mixnet transit that the routing filter does not confine to monitor destinations; and agents share source IPs through distinct host ports while address pools recycle them, so an IP-keyed exemption can be inherited by an unrelated workload. The Noise-authenticated static key that the separately-identified follow-up applies to the mixnet gates CANNOT serve here, because the client websocket port performs no Noise handshake. + +The agent's client identity is permanently recognisable to a gateway. This is accepted: the agent's addresses are published on-chain, so liveness traffic is not covert regardless of the identity's lifetime. Rotating the noise key rotates the derived identity and therefore requires re-announcement. + +#### Scenario: No new key file is required +- **WHEN** an agent performs a gateway liveness test +- **THEN** it uses an identity derived from its noise key, and no additional key is generated on disk or provisioned by an operator + +#### Scenario: A derived identity is stable across runs +- **WHEN** the same agent tests the same gateway on two occasions +- **THEN** the same client identity is presented both times, so an on-chain announcement remains valid and at most one stored entry could ever exist for that pair + +#### Scenario: The session exemption follows the identity, not the address +- **WHEN** an authorised agent opens a client session from an egress address that is not among its authorised mixnet addresses, presenting its announced identity +- **THEN** the gateway grants the ephemeral unmetered session, because the gate is the verified identity + +#### Scenario: An unannounced identity is metered like any client +- **WHEN** a client presents an ed25519 identity that is not in the gateway's authorised-monitor identity set +- **THEN** the session is metered and requires credentials as normal, whatever its source IP + +#### Scenario: A gateway that has not ingested the identity scores the run zero +- **WHEN** an agent tests a gateway that has not yet ingested its announced identity +- **THEN** the session cannot be established without credentials and both phases score zero, which is the same outcome as a missed authorisation event + +### Requirement: nym-api ingests liveness batches on their own endpoint with their own replay state + +nym-api SHALL accept liveness result batches on an endpoint distinct from the stress-testing batch endpoint, applying the same ordered validation (staleness window, contract membership of the signer, strict per-signer timestamp monotonicity, ed25519 signature over the JSON body) and the same `SignedMessage` envelope shape. + +The per-signer high-water mark used for the monotonicity check MUST be held SEPARATELY per endpoint. It MUST NOT be shared with the stress-testing endpoint, because a single orchestrator identity submits both streams and two interleaved streams validated against one high-water mark would reject each other indefinitely. + +Unlike stress-test ingest, liveness ingest MUST accept results for gateway-capable nodes as well as mixnodes, and MUST record each result's per-signal breakdown alongside its averaged score. Rows MUST deduplicate at the database on `(testrun_id, submitter_pubkey)` so that at-least-once resends are idempotent. + +#### Scenario: Interleaved streams do not reject each other +- **WHEN** one orchestrator submits a stress batch and then a liveness batch whose timestamp is lower than the stress batch's +- **THEN** both are accepted, because each endpoint tracks that signer's high-water mark independently + +#### Scenario: Gateway liveness results are accepted +- **WHEN** a liveness batch contains a result for a gateway-only node +- **THEN** it is validated and stored, rather than being dropped as a non-mixnode entry + +#### Scenario: A resent liveness result does not duplicate rows +- **WHEN** the same `(testrun_id, submitter_pubkey)` arrives twice +- **THEN** the second insert is ignored + +### Requirement: Each test kind defines which node types it assigns and how their results are typed + +Each test kind SHALL declare the node types it is eligible to assign. The `stress` kind MUST assign only nodes whose type is `mixnode` or `mixnode_and_gateway` and MUST record its runs as the mixnode test type. The `liveness` kind MUST assign nodes of type `mixnode`, `gateway`, or `mixnode_and_gateway`, selecting the mixnode probe for mixing-capable nodes and the two-phase gateway probe for gateway-capable ones. A node that is both MUST be eligible for both probes, each producing its own signal, and its liveness score MUST be the average over the signals its probes produce. + +A node whose type is `unknown` (never successfully self-described) MUST remain ineligible for every kind. + +#### Scenario: A gateway-only node is assignable for liveness but not for stress +- **WHEN** the orchestrator selects work for a gateway-only node +- **THEN** it may assign a liveness test and never a stress test + +#### Scenario: A dual-role node is measured in both roles +- **WHEN** a `mixnode_and_gateway` node is liveness-tested +- **THEN** it is probed both as a mixing hop and as a gateway, and its score averages every signal produced + +#### Scenario: An unclassified node is never assigned +- **WHEN** a node has never answered its self-description +- **THEN** no test of any kind is assigned to it + +### Requirement: Orchestrator state is a per-kind SQLite schema and the agent registry is in-memory only + +The orchestrator SHALL persist state in a SQLite database whose work-tracking tables are keyed per test kind: a submission-watermark table (one row per kind); `nym_node` (the node registry with its self-described keys, type, announced address set, and gateway client websocket details); a per-kind work-state table keyed `(node_id, test_kind)` holding that kind's last-tested timestamp, last testrun id, and address rotation pointer; `testrun` (completed runs, each recording its kind and which address was tested); a per-signal child table of `testrun` holding the counts and latency distributions of each measured signal; and `testrun_in_progress` (the in-flight dispatch lock set, keyed by `node_id` alone so that only one test of any kind runs against a node at a time, and carrying a materialised `expires_at`). + +The per-kind last-tested timestamp MUST be stored directly rather than read through a join onto the last testrun row, so that evicting an old result does not make a node read as never-tested and jump the assignment queue. + +The agent registry MUST NOT be persisted; it lives only in the in-memory `KnownAgents` cache and is rebuilt from the contract on each startup, which means agents' announced flags reset across a restart and each agent re-announces (and is re-authorised on-chain) on its next run. + +Rehydrating that cache from the contract requires recovering which pair of on-chain entries belongs to one agent. The contract stores one entry per socket address and carries no field linking an agent's two addresses, so the orchestrator MUST group the entries by their x25519 noise key, which is unique per agent (see the network-monitors-contract capability, which does NOT enforce that uniqueness). Entries that do not form exactly one ipv4/ipv6 pair MUST be dropped from the cache rather than guessed at - they are either authorisations predating the paired announcement or leftovers from an agent that has since changed an address - which is safe precisely because the cache only exists to skip redundant contract transactions, and an agent always announces before requesting work. + +#### Scenario: Each kind keeps its own staleness and rotation position +- **WHEN** a node has been tested by both kinds +- **THEN** the work-state table holds one row per kind, each with its own last-tested timestamp and rotation pointer + +#### Scenario: Evicting an old result does not reset staleness +- **WHEN** a node's last completed testrun is deleted by result eviction +- **THEN** the node's per-kind last-tested timestamp is unchanged, so it does not read as never-tested + +#### Scenario: An agent's two on-chain entries are re-paired after a restart +- **WHEN** the orchestrator restarts and reads its agents from the contract +- **THEN** the ipv4 and ipv6 entries sharing one noise key are rehydrated as a single announced agent + +#### Scenario: An unpairable on-chain entry is dropped rather than guessed +- **WHEN** an agent has an on-chain entry with no counterpart of the other family under the same noise key +- **THEN** it is left out of the rehydrated cache and re-created by that agent's next announcement, at the cost of one redundant authorisation transaction + +#### Scenario: Node registry and results survive a restart +- **WHEN** the orchestrator restarts +- **THEN** its node registry, per-kind work state, completed testruns with their signals, and per-kind submission watermarks are loaded from SQLite + +#### Scenario: The agent set is rebuilt from the contract, not from disk +- **WHEN** the orchestrator restarts +- **THEN** its agent registry is rehydrated from the network-monitors contract rather than read from local storage + +## MODIFIED Requirements + +### Requirement: The node refresher builds the testable-node registry from the mixnet contract and each node's self-description + +The node refresher SHALL source the node list from the MIXNET contract (all `NymNodeBond`s), NOT from nym-api. For each bonded node it MUST query that node's self-described HTTP endpoint directly (with host-info verification) to learn EVERY ip address the node announces, its announced mix port, its versioned x25519 noise key, its sphinx key and key-rotation id, and its role-derived `NodeType`. For a node that announces an entry-gateway interface it MUST additionally learn that interface's plain client websocket port, and MUST record whether the node also announces a wss entry (a hostname plus a wss port), because the presence of a wss entry is what distinguishes divergence this subsystem knowingly introduces from divergence that indicates a fault. Per-node queries MUST be bounded by `node_info_query_timeout` (default 10 seconds) and run with concurrency `number_of_concurrent_node_queries` (default 32); a node that fails to answer leaves the corresponding fields NULL. The refresher MUST persist ALL bonded nodes, including unreachable ones (upserting on `node_id`, updating every field except `identity_key`), so that previously-learned keys are retained when a node is transiently unreachable. + +The announced address set MUST be canonicalised (`IpAddr::to_canonical()`), deduplicated and sorted before being stored, because test runs rotate through it by position: a node is free to report its addresses in a different order on every refresh (a resolved hostname typically will), and a duplicate entry would stall the rotation on a subset of the set. The stored `mixnet_socket_address` MUST be derived deterministically from the first address of that sorted set plus the announced mix port, and contributes only that port to the address a given run actually targets. + +#### Scenario: A reachable node's keys are recorded +- **WHEN** the refresher queries a bonded node that answers its self-description +- **THEN** the node's announced address set, socket address, noise key, sphinx key, key-rotation id, and type are stored + +#### Scenario: A gateway's client websocket port is recorded +- **WHEN** the refresher queries a node that announces an entry-gateway interface +- **THEN** the node's plain client websocket port is stored, along with whether it also announces a wss entry + +#### Scenario: The announced address set is stored in a stable order +- **WHEN** a node reports its announced addresses in a different order on a later refresh +- **THEN** the stored set is unchanged, because it is canonicalised, deduplicated and sorted before storage, keeping the per-address test rotation stable + +#### Scenario: An unreachable node is retained with prior data +- **WHEN** a bonded node does not answer within `node_info_query_timeout` +- **THEN** the node row is still upserted, leaving newly-unknown fields NULL and keeping any previously stored keys + +### Requirement: Testruns are assigned lazily from a staleness-ordered node table guarded by an in-flight lock set + +There SHALL be no in-memory work queue. Work is identified by `(node_id, test_kind)`, and staleness, the address rotation and the eligibility gates are all evaluated PER KIND, so that kinds running at different cadences do not disturb one another. + +When an agent requests work, the orchestrator MUST choose the kind, then select targets inside a `BEGIN IMMEDIATE` write transaction that: excludes any node with a `testrun_in_progress` row, REGARDLESS of which kind that row belongs to; requires the fields that kind needs to be non-null; requires the node's type to be one the kind may assign; treats a node as eligible only if that kind has never tested it or last tested it before `now - staleness_age` for that kind; for the `liveness` kind additionally requires that the node's `stress` kind last ran before `now - liveness_after_stress_cooldown`; orders by that kind's test timestamp ascending with never-tested first; takes one target for a `stress` assignment or up to `liveness_wave_size` targets for a `liveness` assignment; rotates each selected node onto the next address in its announced set FOR THAT KIND; records that address as the node's per-kind rotation pointer; and atomically inserts a `testrun_in_progress` row for each, stamped with `started_at`, the kind, and an `expires_at` of `now` plus that kind's lease budget. The response MUST carry the chosen kind and its per-target payload, or an empty assignment when no eligible node exists. + +Excluding any node that has an open in-progress row of ANY kind is required, not incidental: a node being stress-tested at high rate while a liveness probe measures it would bias both results. + +The rotation MUST take the address following the previously handed-out one for that kind, wrapping around at the end of the set and restarting from the first address when the pointer is unset or no longer announced. It MUST advance when the assignment is handed out rather than when a result arrives, so a run that is abandoned still moves the node onto its next address. A node stored before the announced set was tracked MUST remain testable by falling back to the single address in its `mixnet_socket_address`. + +The staleness gate is per NODE AND KIND while the rotation is per ADDRESS, so a node announcing N addresses has each individual address tested by a given kind roughly every N × that kind's `staleness_age` rather than every `staleness_age`. + +#### Scenario: The oldest-tested eligible node is assigned for the chosen kind +- **WHEN** an authorised, announced agent requests a testrun and eligible nodes exist +- **THEN** the orchestrator picks a kind and returns the never-tested-or-oldest node for that kind, inserting a `testrun_in_progress` row for it in the same transaction + +#### Scenario: A node under one kind of test is not assigned another +- **WHEN** a node has an open `testrun_in_progress` row from a stress test +- **THEN** it is excluded from liveness assignment until that row is cleared, and vice versa + +#### Scenario: Kinds do not disturb each other's rotation +- **WHEN** a liveness test and a stress test are both assigned for one node over time +- **THEN** each kind advances its own rotation pointer, so neither skips addresses because of the other + +#### Scenario: A recently stress-tested node is not immediately liveness-tested +- **WHEN** a node's stress test completed more recently than `liveness_after_stress_cooldown` +- **THEN** it is not eligible for a liveness assignment yet, so its liveness score is not measured while it is still recovering from load + +#### Scenario: A liveness assignment carries a wave +- **WHEN** an agent is assigned liveness work and many nodes are eligible +- **THEN** up to `liveness_wave_size` targets are returned in one assignment, each with its own in-flight row and lease + +#### Scenario: No eligible node yields an empty assignment +- **WHEN** every node is either in progress or was tested by the chosen kind more recently than its `staleness_age` +- **THEN** the agent receives an empty assignment and exits without testing + +### Requirement: The orchestrator authorises both of an announcing agent's addresses on-chain in one transaction + +An agent SHALL announce a PAIR of mixnet socket addresses, one ipv4 and one ipv6, because a tested node sees whichever family it was reached over as the source of the probe traffic and gates on that source ip; authorising only one family would leave probes over the other rejected. An agent SHALL additionally announce the base58 ed25519 CLIENT IDENTITY public key it will present when opening a gateway client session, derived from its noise key rather than provisioned, so that the gateway session exemption can be keyed on a verified identity instead of a source address. + +On `POST /v1/agent/announce` the orchestrator SHALL reject with a 400, before touching any state, an announcement whose addresses are not one plain ipv4 address and one ipv6 address that is not ipv4-mapped. Such a pair MUST NOT be normalised into shape, because an ipv4-mapped ipv6 address collapses onto the ipv4 one when a node canonicalises the authorised set, leaving the agent with a single authorised ingress while both the contract and the orchestrator believe it has two, and because rewriting an address would authorise something the agent never announced and will not use in its sphinx return hop. An announced identity key that is not valid base58 decoding to 32 bytes MUST likewise be rejected with a 400 at the same point. + +It MUST then upsert the agent into its in-memory `KnownAgents` cache, keyed by the agent's ipv4 mixnet socket address with the ipv6 address and the identity key held inside the entry, and, if the agent was not already announced, MUST authorise BOTH addresses in the network-monitors contract by submitting ONE transaction carrying an `AuthoriseNetworkMonitor` message per address, each with the agent's base58 x25519 noise key, noise version, and identity key, then mark the agent announced. Both authorisations MUST travel in a single transaction so that an agent is never left with only one of its addresses authorised. A contract transaction failure MUST surface as a 500 and leave the agent un-announced; re-announcing is safe because the contract's agent save is an upsert. An agent whose announced noise key, ipv6 address, OR identity key differs from the cached one MUST have its announced flag reset so it is re-authorised, and that divergence SHOULD be surfaced (log plus counter) because a superseded ipv6 address stays authorised in the contract. This on-chain write is what ultimately causes network nodes to accept the agent's probe connections and to recognise its client sessions. + +Because the identity is carried on an OPTIONAL contract field and the save is an upsert, agents authorised before the field existed acquire it on their next announcement with no backfill step. The orchestrator MUST NOT treat a cached entry without an identity as invalid, since it may have been rehydrated from such an entry after a restart. + +#### Scenario: A first announcement authorises both addresses on-chain +- **WHEN** a not-yet-announced agent calls `announce` +- **THEN** the orchestrator writes a single transaction carrying one `AuthoriseNetworkMonitor` message for each of the agent's two addresses, both with its noise key and its identity key, and marks it announced + +#### Scenario: A contract failure is not silently swallowed +- **WHEN** the authorisation transaction fails +- **THEN** the announce call returns a 500 and the agent remains un-announced, with neither address authorised + +#### Scenario: A malformed address pair is rejected outright +- **WHEN** an agent announces two addresses of the same family, the same address twice, or an ipv4-mapped ipv6 address +- **THEN** the call is rejected with a 400 and nothing is cached or written on-chain + +#### Scenario: A malformed identity key is rejected outright +- **WHEN** an agent announces an identity key that is not valid base58 decoding to 32 bytes +- **THEN** the call is rejected with a 400 and nothing is cached or written on-chain + +#### Scenario: A changed identity key triggers re-authorisation +- **WHEN** an already-announced agent announces an identity key that differs from the cached one +- **THEN** its announced flag is reset, both addresses are authorised again carrying the new identity, and the divergence is logged and counted + +### Requirement: Network nodes learn the authorised-agent set from the contract and gate connection, routing, and replay-bypass on it + +A Nym node SHALL derive which network-monitor agents may probe it directly from the network-monitors contract, not from any orchestrator or nym-api. A node MUST load the full authorised-agent set once at startup (via `get_all_network_monitor_agents`; a failed load aborts node startup) and MUST thereafter keep it current in REAL TIME through a nyxd websocket event subscription that dispatches `AuthoriseNetworkMonitor`, `RevokeNetworkMonitor`, and `RevokeAllNetworkMonitors` contract events. This is an event subscription, NOT a periodic contract poll; the node's periodic topology refresher explicitly preserves (does not reload) the agent set. + +The node MUST fold the set into THREE shared, lock-free structures: a canonical-IP-keyed routing set (`RoutableNetworkMonitors`), a canonical-IP-keyed noise-key map (`NoiseNetworkView`, in which one IP may host several agents disambiguated by port), and a set of announced monitor ed25519 CLIENT IDENTITIES keyed by that identity rather than by any address. The two IP-keyed structures MUST key on `IpAddr::to_canonical()` at insert AND lookup so that a v4-mapped-IPv6 form matches its canonical IPv4 form. The identity set MUST tolerate the same identity arriving from several agent entries, since an agent authorises one entry per address family and both carry its identity, and MUST tolerate entries carrying no identity at all, which is a validly authorised agent that simply cannot be recognised on the client-session path. There is no separate "extra initiator IPs" allowlist; inbound acceptance is a facet of the noise map. The authorised set MUST gate five behaviours: (1) the Noise responder handshake - an inbound connection from an IP not in the noise map falls back to raw TCP and the agent's handshake fails; (2) packet routing through `NetworkRoutingFilter`, in which a packet originating from an authorised monitor may ONLY be routed to another authorised monitor; (3) most importantly, the sphinx REPLAY / bloomfilter BYPASS - a packet detected as replayed MUST be dropped as a replay UNLESS it originates from an authorised network-monitor agent IP, which is the mechanism that lets the agent's deliberately-replayed probe header (see the `reuse_header` requirement) be processed rather than filtered; (4) FINAL-HOP DELIVERY - a final-hop packet originating from an authorised monitor MUST be processed and delivered to a live client session, and MUST NOT be written to the recipient's on-disk store if no session is live, so that a packet which did not arrive on the socket was definitively not delivered and monitor traffic cannot accrue undeliverable stored messages on every gateway; and (5) CLIENT SESSION METERING - a client websocket session whose registration handshake authenticates an ed25519 identity in the announced-identity set MUST be treated as an ephemeral monitor session: it MUST NOT be metered for bandwidth, MUST NOT require any bandwidth credential, and MUST NOT persist a shared-key, bandwidth, or stored-message entry. + +Gates (4) and (5) are what make gateway liveness testing possible; before them a monitor's final-hop packets were dropped outright as unsupported, and a monitor could not open a client session without presenting bandwidth credentials. + +Gates (1) through (4) are keyed by SOURCE IP only (not public key); the port is effectively ignored on the agent-as-initiator probe path (it is consulted only when the node dials an agent). Gate (5) is the ONE exception and MUST be keyed on the handshake-verified client identity, with the source IP playing no part, because the client websocket port performs no Noise handshake and so can never be covered by the follow-up that moves the mixnet gates onto the Noise-authenticated static key, and because the exemption it guards is not confined by gate (2): a packet handed to a gateway over a client session is forwarded without the monitor flag and may therefore be routed to any known node, so an IP-keyed exemption inherited by a co-tenant behind a shared host port or a recycled address pool would grant unconfined unmetered transit. + +The consequences are: an agent cannot successfully probe a node until it is authorised on-chain AND that authorisation event has been ingested by the node (propagation is bounded by block inclusion plus websocket delivery, on the order of seconds, NOT by any refresh interval); for gates (1) through (4) the IP the agent actually connects from MUST equal one of the `mixnet_address` IPs recorded on-chain for it (an egress IP that is neither, whether through NAT or a third interface, still breaks all four); an authorised agent with an announced identity obtains unmetered gateway transit for sessions presenting that identity, bounded by the orchestrator's ability to revoke the authorisation; and because there is no periodic reconciliation against the contract, a node that misses a revoke event (for example during websocket downtime) only re-syncs on its next restart's one-time load, which for gate (5) means a revoked monitor keeps its unmetered sessions until then. + +Because an agent authorises one ipv4 and one ipv6 address, it occupies TWO entries in each node's structures - one per address, both carrying the same noise key - so a probe arriving over either family passes every gate. The node treats those entries independently: it neither knows nor needs to know that they belong to one agent, and revoking one leaves the other authorised. + +Intended follow-ups (recorded here as planned changes, NOT current behaviour): (1) add a periodic reconciliation of each node's authorised-agent set against the contract, so a missed revoke event no longer lingers until the next node restart - this is a prerequisite for liveness scores ever carrying weight, because a node that missed its agents' authorisation events fails every gate and is indistinguishable from a dead node; and (2) gate the replay bypass and the final-hop delivery on the agent's Noise-authenticated x25519 static key rather than its source IP. The current `Noise_XKpsk3` handshake already receives and possession-authenticates that key (the message-3 `se` step proves the agent holds the corresponding private key), so this hardening needs no packet-format change and would remove the source-IP spoofing and NAT-fragility of the present gates. Follow-up (2) covers the MIXNET gates only; the client-session exemption is out of its reach and is why gate (5) is identity-keyed from the outset. + +#### Scenario: A newly authorised agent is accepted in near real time +- **WHEN** an orchestrator authorises an agent on-chain and the transaction is included in a block +- **THEN** each node's websocket watcher ingests the `AuthoriseNetworkMonitor` event and adds the agent's IP and noise key to its routing set and noise map without waiting for any refresh interval + +#### Scenario: An unauthorised agent cannot complete a handshake or have replays accepted +- **WHEN** an agent that the node has not ingested opens a connection and sends replayed packets +- **THEN** the Noise handshake falls back to raw TCP and fails, and any replayed packet is dropped as a replay because it does not come from an authorised agent IP + +#### Scenario: Replayed probe traffic from an authorised agent bypasses the bloomfilter +- **WHEN** an authorised agent sends its deliberately-replayed probe header +- **THEN** the node still runs its replay-detection bloomfilter but bypasses the drop because the packet's source IP is in the authorised network-monitor set, and processes the packet + +#### Scenario: A monitor's final-hop packet is delivered to its live session +- **WHEN** an authorised agent sends a final-hop packet to a gateway addressed to a client session it currently holds open +- **THEN** the gateway unwraps it and pushes it into that session + +#### Scenario: A monitor's final-hop packet is dropped rather than stored +- **WHEN** an authorised agent sends a final-hop packet whose recipient has no live session +- **THEN** the packet is dropped and nothing is written to the on-disk store + +#### Scenario: A monitor's client session needs no bandwidth +- **WHEN** an authorised agent opens a client websocket session presenting its announced ed25519 identity and forwards packets +- **THEN** the session is not metered, no credential is required, and no shared-key, bandwidth, or stored-message entry is persisted for it + +#### Scenario: An authorised IP alone does not earn the session exemption +- **WHEN** a client opens a websocket session from an authorised agent's IP but presents an identity that is not in the announced-identity set +- **THEN** the session is metered and requires credentials like any other client's + +#### Scenario: Revocation stops acceptance after the event is ingested, with no periodic re-sync +- **WHEN** an agent is revoked on-chain and the node ingests the `RevokeNetworkMonitor` event +- **THEN** the node removes it from the routing set and noise map so new handshakes fail, replays are dropped again, and its client sessions are metered like any other +- **AND** if the node misses that event it will only re-sync the agent set on its next restart, because there is no periodic reconciliation + +### Requirement: Completed testruns are submitted to nym-api in signed, monotonic batches with at-least-once delivery + +The result submitter SHALL forward completed testruns to nym-api, in a SEPARATE STREAM PER TEST KIND. Each stream MUST have its own destination endpoint and its own persisted watermark, because one watermark cannot describe two destinations: advancing a shared watermark for one stream would skip unsubmitted rows of the other. Stress results MUST be submitted to `POST /v3/nym-nodes/stress-testing/batch-submit`; liveness results MUST be submitted to their own endpoint. + +For each stream the submitter MUST read that stream's persisted watermark, fetch completed testruns of that kind after it in ascending id order, and send them in chunks of `result_submission_batch_size` (default 50). Each stress `TestRun` MUST be converted to a `StressTestResult` whose `test_performance` is `packets_received / packets_sent` (or `0.0` when `packets_sent` is zero or duplicates were seen) and whose `was_reachable` is `error.is_none()`. Each liveness `TestRun` MUST be converted to a result whose performance is the average over that kind's fixed signal set, carrying the per-signal breakdown, with a signal that produced no measurement counted as zero. Each batch MUST be wrapped in a submission content carrying `{ signer, timestamp, results }`, given a timestamp that is strictly increasing (bumped by 1 nanosecond if the clock has not advanced since the last batch, matching nym-api's replay guard), and signed with the orchestrator's ed25519 identity key. Each stream's watermark MUST be advanced only AFTER a successful POST, so a failed submission re-sends the same testruns on the next cycle (at-least-once delivery). + +#### Scenario: Only new testruns are submitted, in order +- **WHEN** the submitter runs with a watermark of N for a given kind +- **THEN** it submits testruns of that kind with id greater than N in ascending id order, chunked by the batch size + +#### Scenario: One stream's progress does not advance another's +- **WHEN** a liveness batch is submitted successfully while stress results are still pending +- **THEN** only the liveness watermark advances and the pending stress results are still submitted on their own stream + +#### Scenario: A failed POST is retried, not skipped +- **WHEN** a batch POST fails +- **THEN** that stream's watermark is not advanced and the same testruns are resubmitted on the next cycle + +#### Scenario: Batch timestamps are strictly monotonic +- **WHEN** two batches are produced within the same clock tick +- **THEN** the second batch's timestamp is bumped so it is strictly greater than the first, satisfying nym-api's replay check + +### Requirement: Stale in-flight dispatches and old results are evicted + +The stale-data eviction task SHALL clear `testrun_in_progress` rows whose `expires_at` has passed, so that a dispatch abandoned by a crashed or hung agent frees its node for reassignment, and MUST delete completed testruns older than `testrun_eviction_age` (default 7 days), along with their per-signal rows. One eviction sweep MUST run before the HTTP server begins serving. + +The deadline MUST be materialised on the in-progress row at hand-out rather than derived by the sweep from a single global timeout, because different test kinds have different budgets (a stress run is minutes, a liveness wave is seconds) and a future kind may be more expensive still. The sweep therefore requires no knowledge of kinds. + +#### Scenario: A timed-out dispatch is released +- **WHEN** a `testrun_in_progress` row's `expires_at` has passed +- **THEN** it is removed and the node becomes eligible for assignment again + +#### Scenario: Kinds with different budgets coexist +- **WHEN** a long-budget stress dispatch and a short-budget liveness dispatch are both in flight +- **THEN** each is evicted according to its own deadline, and the short one does not keep the long one alive nor vice versa + +#### Scenario: Old results are pruned +- **WHEN** completed testruns are older than `testrun_eviction_age` +- **THEN** they and their per-signal rows are deleted from the database + +### Requirement: The agent is a one-shot job that announces, requests one assignment, tests, submits, and exits + +The `run-agent` path SHALL be a run-to-completion job, NOT a long-lived daemon: it builds an orchestrator client with a bearer token, loads its x25519 noise key, announces itself, requests a single assignment, and - if one is returned - executes it and exits. An assignment is either ONE stress target or a WAVE of liveness targets executed concurrently; in the wave case each target's result MUST be submitted as soon as that target finishes rather than at the end of the wave. When the assignment is empty it MUST log that no work is available and exit without testing. Fleet scale is therefore achieved by running many short-lived agent invocations rather than one persistent process, with liveness sweep throughput coming from wave concurrency within an invocation rather than from a longer-lived process. + +The agent MUST be able to execute every test kind the orchestrator may assign, since the orchestrator is the party that chooses. The agent binary MUST also provide `build-info`, a `keygen` subcommand that generates ONLY an x25519 noise key (no ed25519 key), and a `test-node` subcommand that runs a single manual test against an explicitly-specified node bypassing the orchestrator (with no `node_id`). + +#### Scenario: An assignment is executed once and submitted +- **WHEN** the agent receives a non-empty assignment +- **THEN** it executes every target it was given, submits each result, and exits + +#### Scenario: No work available exits cleanly +- **WHEN** the agent receives an empty assignment +- **THEN** it logs that no work is available and exits without testing or submitting + +#### Scenario: A wave's results are submitted as they complete +- **WHEN** one target of a liveness wave finishes while others are still running +- **THEN** its result is submitted immediately rather than being held until the wave ends + +### Requirement: The per-node result captures counts, handshake and latency statistics, and an optional error + +Each test SHALL produce a result carrying its test kind, `time_taken`, and an optional `error`, plus ONE OR MORE SIGNALS. Each signal MUST carry: ingress and egress Noise-handshake durations; the sphinx packet delay; `packets_sent` and `packets_received`; the baseline `approximate_latency`; per-packet and per-send latency distributions (minimum, mean, median, maximum, standard deviation); and a `received_duplicates` flag. A stress or mixnode-liveness run produces exactly one signal; a gateway-liveness run produces one per phase. Only a critical failure (for example an inability to bind the ingress listener) MUST bubble up as an error return; node-level failures (no response, bloomfilter misconfiguration, a rejected Noise handshake, a refused client session) MUST be recorded inside the returned result so the orchestrator always receives partial data. + +The per-signal breakdown MUST be persisted and exposed on the operator read surface even though downstream consumers receive only the averaged score, because a gateway with a healthy ingress and a dead egress is otherwise indistinguishable from one that is uniformly half-lossy. + +The submission that carries a result to the orchestrator MUST additionally report WHICH address was tested, and that address MUST be persisted with the run and exposed on the operator read surface. A node may announce several addresses of which only some are healthy, so without it a per-address failure is indistinguishable from a dead node and gets averaged into that node's single result series. + +Latency statistics MUST be recorded but MUST NOT contribute to any score. Two confounds make them unsuitable for scoring today: a node defers replay checking in batches bounded by its deferral time and pending-packet count and only defers when its bloomfilter lock is contended, so a low-volume probe measures a busy node as slower than an idle one in a step function; and measurements taken inside a concurrent wave include the agent's own queueing. Consequently liveness latency figures MUST NOT be compared with stress-test latency figures. + +#### Scenario: A node-level failure still yields a result +- **WHEN** a node fails to respond, is misconfigured, or refuses the Noise handshake +- **THEN** the agent returns a result with the failure recorded in its `error` field rather than failing the job + +#### Scenario: A result is attributable to the address it measured +- **WHEN** a run against one of a node's several announced addresses fails +- **THEN** the stored run records the tested address, so the failure is attributable to that address rather than to the node as a whole + +#### Scenario: A gateway result keeps its per-phase breakdown +- **WHEN** a gateway liveness run is stored and read back +- **THEN** the ingress and egress signals are separately visible, alongside the averaged score submitted downstream + +#### Scenario: A rejected handshake scores zero rather than being excluded +- **WHEN** a node rejects the agent's Noise handshake, so nothing can be measured +- **THEN** the result records the failure and scores zero, because a node that will not accept a connection is not routable and an unmeasurable node must not score better than a measurably broken one + +### Requirement: nym-api accepts batches only from contract-authorised orchestrators after staleness, replay, and signature checks + +The nym-api handler for `POST /v3/nym-nodes/stress-testing/batch-submit` SHALL validate each submission through six ordered steps: (1) reject the batch if its body is older than a 30-second staleness window; (2) reject it unless the signer's ed25519 public key is in the `NetworkMonitorsCache` authorised set, which is populated from the network-monitors contract's authorised-orchestrator identity keys and refreshed lazily on a TTL (default 30 minutes); (3) reject it unless its timestamp is strictly greater than the per-signer high-water mark held in an in-memory `LastNMSubmissions` map, falling back to the process-online time when no prior submission is recorded (for example after a restart); (4) reject it unless the ed25519 signature over the JSON body verifies against the signer; (5) update the per-signer high-water mark; and (6) validate and insert the individual results. + +The per-signer high-water mark MUST be scoped to the submission endpoint. Each ingest endpoint MUST keep its own map, so that two streams signed by one orchestrator identity cannot invalidate each other's timestamps. + +Because the per-signer high-water mark is held in memory, it resets to the process-online time on restart; the database primary-key dedupe described in the next requirement is what ultimately guarantees idempotency. Intended follow-up (recorded here as a planned change, NOT current behaviour): persist the per-signer high-water mark across restarts as defense-in-depth. + +#### Scenario: A batch from an unknown signer is rejected +- **WHEN** the signer's key is not in the contract-derived authorised set +- **THEN** the submission is rejected as unauthorised + +#### Scenario: A replayed or out-of-order batch is rejected +- **WHEN** a batch's timestamp is not strictly greater than the signer's last accepted timestamp for that endpoint +- **THEN** the submission is rejected + +#### Scenario: A tampered batch fails the signature check +- **WHEN** the body does not match its ed25519 signature for the given signer +- **THEN** the submission is rejected as failing its integrity check + +#### Scenario: One endpoint's high-water mark does not gate another's +- **WHEN** the same orchestrator submits to the stress and liveness endpoints with interleaved timestamps +- **THEN** each endpoint evaluates monotonicity against its own map and both submissions are accepted + +### Requirement: Stored stress-test scores feed node performance and rewarding through a defined consumer surface + +The stored stress-test and liveness results SHALL form the subsystem's output contract to the rest of nym-api; the detailed behaviour of each consumer is owned by its own capability, and this requirement fixes only WHICH subsystems read the results and FOR WHAT. The stored per-node stress results MUST be aggregated (average performance and a reachability flag over a configured window) into a stress-testing score; the stored per-node liveness results MUST be aggregated the same way into a separate liveness score; each score MUST feed the node performance provider, which folds them - together with routing and configuration components - into each node's detailed performance, each gated by its own `use_*_data`, `minimum_available_*_results`, and `*_score_weight` configuration flags; and the resulting composite performance MUST flow into rewarding via the node's rewarding-performance derivation. + +The liveness score's weight MUST default to ZERO, so that liveness is recorded and queryable without affecting performance or rewarding until the operator deliberately enables it. This is required because two populations will score zero on liveness for reasons unrelated to their forwarding capability: nodes that have not ingested their agents' on-chain authorisations, and gateways not yet carrying the final-hop and monitor-session behaviour that gateway liveness depends on. + +While the weight is zero, nym-api MUST expose a DIVERGENCE metric comparing each node's aggregated liveness score against the v1 monitor's routing score, bucketed by whether the node announces a wss entry gateway address. The bucketing is required because this subsystem deliberately probes only the plain-ws ingress, so a gateway with a broken TLS ingress is EXPECTED to diverge; without the bucket, that expected divergence is indistinguishable from a node that never learned about its agents. This metric is the evidence on which the eventual decisions to weight liveness, and separately to retire the v1 routing score, are to be based. + +#### Scenario: Stress scores contribute to node performance when enabled +- **WHEN** stress-testing data is enabled and a mixnode has at least the minimum number of available results +- **THEN** its averaged stress score is folded into its detailed performance according to the configured weight + +#### Scenario: Liveness scores are inert by default +- **WHEN** liveness results are stored and aggregated with the default configuration +- **THEN** they are queryable and appear in the divergence metric, but contribute nothing to any node's performance or reward + +#### Scenario: Divergence is attributable +- **WHEN** a gateway scores zero on liveness while the v1 monitor scores it as routable +- **THEN** the divergence metric records it in the bucket matching whether it announces a wss entry, so an expected TLS-path divergence is distinguishable from an unexpected one + +#### Scenario: The consumer surface is bounded +- **WHEN** reasoning about the blast radius of a stress or liveness score +- **THEN** the readers are the performance aggregation query, the performance provider, and rewarding, each specified by its own capability + +### Requirement: The subsystem's behaviour is governed by orchestrator and agent configuration surfaces with defined defaults + +The orchestrator SHALL be configured with the following defaults: `test_interval` 2 hours, `test_timeout` 5 minutes, `node_refresh_rate` 2 hours, `node_info_query_timeout` 10 seconds, `testrun_eviction_age` 7 days, `result_submission_interval` 15 minutes, `result_submission_batch_size` 50, `number_of_concurrent_node_queries` 32, `chain_authorisation_check_max_attempts` 10, `chain_authorisation_check_retry_delay` 1 minute, and an HTTP bind of `0.0.0.0:8080`; plus required secrets (`agents_token`, `metrics_and_results_token`, the bip39 `mnemonic`, and the base58 ed25519 `private_key`) and required endpoints (`nym_api_endpoint`, `rpc_url`, the mixnet and network-monitors contract addresses, and `database_path`). + +Where a knob governs a per-kind behaviour it MUST be expressible per kind. The orchestrator MUST additionally carry, for the liveness kind: a staleness interval (defaulting well below the stress `test_interval`, so that liveness tracks v1's cadence), a lease budget used as the in-progress `expires_at` (which MUST bound one concurrent wave, not the sum over its targets), a wave size, a `liveness_after_stress_cooldown`, and an enable flag allowing liveness assignment to be switched off without redeploying. `test_timeout` remains the stress kind's lease budget. + +The agent SHALL be configured with the following defaults: `sending_duration` 30 seconds, `waiting_duration` 5 seconds, `packet_delay` 50 milliseconds (which MUST be non-zero), `target_rate` 1000 packets/second, `reuse_header` true, `egress_connection_timeout` 5 seconds, `noise_handshake_timeout` 3 seconds, `sending_batch_size` 50, and a listener bind of `[::]:9000`; plus the required orchestrator URL, orchestrator bearer token, announced ipv4 host address, announced ipv6 host address, shared announced port, and noise-key path. The agent MUST additionally carry a liveness profile: a per-target packet count, an AGGREGATE send-rate budget from which the per-target rate is derived (never the reverse), a straggler wait, and per-target timeouts. All knobs MUST be overridable by CLI flag or environment variable. + +The liveness profile's initial values are PROVISIONAL, chosen for score granularity rather than measured against agent hardware: a per-target packet count of 100 (giving 1% granularity against v1's three packets per route), an aggregate budget of 500 packets/second, and a wave size of 20. Because they are provisional, every one of them MUST be tunable in a deployment without a code change, and no behaviour may depend on a specific value: an agent host that cannot sustain the aggregate budget MUST be correctable by configuration alone. + +The announced pair MUST be validated at configuration time, applying the same rule the orchestrator enforces on announce, so a misconfigured deployment fails immediately rather than on its first announcement. The listener bind default MUST remain dual-stack (`[::]`), since an ipv4-only bind cannot receive the return traffic for a run whose return hop is the agent's ipv6 address, and since one shared listener serves every target of a concurrent wave. + +#### Scenario: Defaults match the documented values +- **WHEN** an orchestrator or agent is configured without overriding a given knob +- **THEN** the effective value is the default listed above + +#### Scenario: A zero packet delay is rejected +- **WHEN** the agent is configured with a `packet_delay` of zero +- **THEN** configuration construction fails + +#### Scenario: A malformed announced address pair is rejected at startup +- **WHEN** the agent is configured with two announced addresses of the same family, or with an ipv4-mapped ipv6 address +- **THEN** configuration construction fails before the agent announces itself + +#### Scenario: Liveness can be disabled without a redeploy +- **WHEN** the orchestrator's liveness enable flag is unset +- **THEN** no liveness assignment is handed out and stress testing continues unaffected + +## REMOVED Requirements + +### Requirement: Orchestrator state is a four-table SQLite database and the agent registry is in-memory only + +**Reason**: The schema is no longer four tables, and its shape is no longer describable per node rather than per (node, kind). Work state moves into a per-kind table, results gain a per-signal child table, in-progress rows gain a materialised lease, and the single submission watermark becomes one per kind. The requirement's normative content is replaced by "Orchestrator state is a per-kind SQLite schema and the agent registry is in-memory only", which restates the agent-rehydration rules unchanged. + +**Migration**: A migration MUST move `nym_node.last_testrun` and `nym_node.last_tested_ip` into the new per-kind work-state table under the `stress` kind, so existing nodes keep their staleness position and address rotation; MUST backfill `expires_at` on any in-progress row from `started_at` plus the stress lease budget; and MUST carry the existing `metadata.last_submitted_testrun_id` across as the stress stream's watermark. No result data is lost: existing `testrun` rows keep their columns and are read as single-signal runs. + +### Requirement: The subsystem tests mixnodes only; the gateway test path is an unwired extension seam + +**Reason**: Gateways are now tested by the liveness kind, so the statement that the gateway path is unwired scaffolding is no longer true. Replaced by "Each test kind defines which node types it assigns and how their results are typed", which keeps the mixnodes-only restriction for the stress kind and defines gateway eligibility for the liveness kind. + +**Migration**: None for stored data: stress runs continue to be recorded as the mixnode test type and nym-api continues to drop non-mixnode entries on the stress endpoint. Gateway results arrive only on the liveness endpoint, which accepts them. diff --git a/openspec/changes/network-monitor-liveness-tests/tasks.md b/openspec/changes/network-monitor-liveness-tests/tasks.md new file mode 100644 index 00000000000..3df41cf3e7f --- /dev/null +++ b/openspec/changes/network-monitor-liveness-tests/tasks.md @@ -0,0 +1,114 @@ +## 1. Provisional defaults and the one compatibility check + +- [ ] 1.1 Adopt the provisional liveness profile defaults (100 packets per target, a 500 packets/second aggregate budget, a wave size of 20) and make every one of them CLI- and env-overridable, so an agent host that cannot sustain the budget is a configuration change rather than a code change +- [ ] 1.2 Size the liveness lease budget, straggler wait and per-target timeouts from those defaults with slack, and make them configurable on the same terms, with no behaviour depending on a specific value +- [ ] 1.3 Confirm on a devnet that an un-upgraded node still ingests an `AuthoriseNetworkMonitor` carrying an unknown field, rather than logging a parse failure and skipping it. The `cw_serde` reading says it will; this assumption is load-bearing for every future contract change and gates the orchestrator deploy, not the migration, which emits no message at all + +## 2. Contract and announcement: the agent ed25519 identity + +- [ ] 2.1 Add an optional `bs58_ed25519_identity` to `ExecuteMsg::AuthoriseNetworkMonitor` and to the stored `AuthorisedNetworkMonitor` in `common/cosmwasm-smart-contracts/network-monitors-contract`, as an added field on the existing variant (never a new variant) +- [ ] 2.2 Validate the identity on shape in `try_authorise_network_monitor` (base58 decoding to exactly 32 bytes) with its own error variant, and accept its absence without complaint +- [ ] 2.3 Bump the contract version, leave `migrate` as build-information only, and leave `queued_migrations` empty: the field is optional and the agent save is an upsert, so there is no data migration +- [ ] 2.4 Regenerate the contract JSON schema +- [ ] 2.5 Extend the validator-client signing helper so the authorisation message carries the identity +- [ ] 2.6 Contract tests: an omitted identity is accepted, a malformed one is rejected, a re-authorisation records a changed identity, and an entry serialised without the field deserialises with `None` +- [ ] 2.7 Add a regression test asserting that a serialised new-form `AuthoriseNetworkMonitor` still deserialises into a struct shaped like the old one, so the fleet-compatibility assumption behind this change is checked in CI rather than assumed +- [ ] 2.8 Agent side: derive the ed25519 identity from the x25519 noise private key via a labelled HKDF whose output is the ed25519 seed, and include its base58 public key in the announce request +- [ ] 2.9 Orchestrator side: carry the identity on the announce request and in both `AuthoriseNetworkMonitor` messages of the existing single transaction, reject a malformed identity with a 400 before touching state, hold it in the `KnownAgents` entry, and reset the announced flag when it diverges from the cached one +- [ ] 2.10 Tolerate a rehydrated cache entry that has no identity, since it may come from an entry authorised before the field existed +- [ ] 2.11 Add a counter for identity divergence alongside the existing agent-details-changed counter +- [ ] 2.12 Prepare the multisig migration proposal, noting that the only consumer of the new field is a gateway that must be upgraded for task group 7 regardless +- [ ] 2.13 Unit-test that an announcement with a malformed identity is rejected before any cache write or contract call, and that a changed identity re-authorises both addresses + +## 3. Shared request/response types + +- [ ] 3.1 Add a `TestKind` enum (`stress`, `liveness`) to `nym-network-monitor-orchestrator-requests` +- [ ] 3.2 Replace `TestRunAssignment` with a kind-tagged assignment carrying per-kind payloads: stress and mixnode-liveness (node address, node ips, noise key, sphinx key, key rotation id, probe profile) and gateway-liveness (additionally the client websocket port and the gateway identity key) +- [ ] 3.3 Make the assignment response carry a wave (a list of targets) for liveness and a single target for stress, with an empty response still meaning "no work" +- [ ] 3.4 Add a per-signal result shape (`signal` discriminator plus the existing counts, handshake durations and latency distributions) and make the submission request carry the run-level fields plus one or more signals +- [ ] 3.5 Add the liveness submission route constant and the liveness batch content type mirroring the stress `SignedMessage` envelope +- [ ] 3.6 Unit-test that a gateway-liveness run serialises and round-trips both of its signals, and that a single-signal run round-trips unchanged + +## 4. Orchestrator storage and migration + +- [ ] 4.1 Write migration `03`: create the per-kind work-state table keyed `(node_id, test_kind)` holding `last_tested_at`, `last_testrun_id`, `last_tested_ip`; create the per-signal child table of `testrun`; create the per-kind submission-watermark table; add `test_kind` to `testrun`; add `expires_at` and `test_kind` to `testrun_in_progress`; add the gateway client websocket port and the wss-announced flag to `nym_node` +- [ ] 4.2 In the same migration, backfill: move `nym_node.last_testrun` / `last_tested_ip` into the work-state table under the `stress` kind, derive `expires_at` for any live in-progress row from `started_at` plus the stress budget, carry `metadata.last_submitted_testrun_id` across as the stress stream's watermark, and project existing `testrun` rows as single-signal runs +- [ ] 4.3 Drop `nym_node.last_testrun` and `nym_node.last_tested_ip` once the backfill is in place +- [ ] 4.4 `touch build.rs` and confirm the compiled-in migrations DB picks up the new columns (a "table X has no column named Y" error means the build script did not re-run) +- [ ] 4.5 Update `storage/models.rs`: per-kind work-state row, per-signal row, `TestKind` mapping, and move the address rotation helpers (`announced_ips`, `next_ip_to_test`) onto the per-kind state so a kind rotates independently +- [ ] 4.6 Update the storage manager's insert path to write a run-level row plus its signal rows in one transaction, and the read paths to reassemble them +- [ ] 4.7 Update result eviction to delete signal rows with their run, and in-progress eviction to compare `expires_at` rather than a global cutoff +- [ ] 4.8 Unit-test the per-kind rotation (two kinds advancing independently over one node's address set) and that deleting a node's last testrun leaves its per-kind `last_tested_at` intact + +## 5. Orchestrator scheduling + +- [ ] 5.1 Rewrite `assign_next_mixnode_testrun` as a kind-aware assignment: choose the kind, filter by that kind's eligible node types and required non-null fields, apply that kind's staleness age, exclude any node with an in-progress row of any kind, apply `liveness_after_stress_cooldown` for liveness, take one target for stress or up to `liveness_wave_size` for liveness, advance each node's per-kind rotation pointer, and insert one in-progress row per target with its lease +- [ ] 5.2 Add the kind-selection policy (which kind an agent is handed when several are due) and the liveness enable flag that switches liveness assignment off without a redeploy +- [ ] 5.3 Extend the node refresher to record the entry-gateway client websocket port and whether a wss entry is announced +- [ ] 5.4 Add the liveness config knobs (staleness interval, lease budget, wave size, cooldown, enable flag) with the provisional defaults from 1.1 and 1.2, all CLI- and env-overridable +- [ ] 5.5 Add prometheus series for liveness assignments, wave sizes, per-kind in-progress counts, lease expiries, and cooldown skips +- [ ] 5.6 Unit-test that a node with an open stress in-progress row is not assigned liveness and vice versa, that a recently stress-tested node is skipped by the cooldown, and that a wave never exceeds `liveness_wave_size` + +## 6. nym-node: final-hop delivery for monitors + +- [ ] 6.1 Replace the unconditional drop of network-monitor final-hop packets in `handle_final_hop` with delivery to a live client session +- [ ] 6.2 Suppress the on-disk fallback for network-monitor final-hop packets: when no session is live, drop and count the packet rather than storing it +- [ ] 6.3 Add metrics distinguishing a monitor final-hop packet delivered in-session from one dropped for want of a session +- [ ] 6.4 Unit-test both branches of 6.1 and 6.2, asserting that nothing is written to the store on the drop path + +## 7. nym-node: ephemeral unmetered monitor client session + +- [ ] 7.1 Derive a third structure from the authorised-agent set: the announced monitor ed25519 identities, keyed by identity rather than by address, populated from the same startup load and the same nyxd websocket events, tolerating entries with no identity and the same identity arriving from both of an agent's entries +- [ ] 7.2 Add an `ephemeral` mode to `BandwidthStorageManager` that seeds a synthetic allowance and performs no read or write against `BandwidthGatewayStorage` +- [ ] 7.3 Make the client identity threaded into `ClientDetails` / `BandwidthStorageManager` optional (or a `Persisted` / `Ephemeral` discriminator) so a session with no storage row is representable +- [ ] 7.4 Route a client websocket session into an ephemeral monitor session when the registration handshake's verified ed25519 identity is in the set from 7.1, skipping `insert_shared_keys`, `create_bandwidth_entry`, and the stored-message push. The source IP MUST play no part in this decision +- [ ] 7.5 Ensure an out-of-bandwidth outcome is reported to the client as a distinguishable error rather than a generic failure, so a proxied gateway or a gateway that has not ingested the identity is diagnosable +- [ ] 7.6 Unit-test that a monitor session forwards packets without a credential and leaves the gateway storage untouched, that a session presenting an unannounced identity from an authorised agent IP is metered exactly as before, and that a session presenting an announced identity from an unrelated IP is exempt + +## 8. Agent: liveness profile and wave concurrency + +- [ ] 8.1 Add the liveness probe profile alongside the stress profile in the tester config, with the aggregate rate budget deriving the per-target rate +- [ ] 8.2 Bind ONE shared ingress listener per invocation and build a `NoiseNetworkView` containing every target's noise key under every address that target is known by +- [ ] 8.3 Make the known-source set the union of every target's announced addresses, canonicalised +- [ ] 8.4 Attribute returned packets to a target by the source address of the connection they arrive on, and accumulate per-target results +- [ ] 8.5 Execute a wave as one concurrent batch with a hard per-target deadline, so the wave's duration is bounded by the slowest single target +- [ ] 8.6 Submit each target's result as soon as that target finishes rather than at the end of the wave +- [ ] 8.7 Unit-test attribution across a wave (several targets returning interleaved packets), and that one target timing out does not extend the others + +## 9. Agent: gateway client session and the two-phase probe + +- [ ] 9.1 Use the derived ed25519 client identity from 2.8 for the session, with no new on-disk key and no per-test regeneration (the identity must match what was announced) +- [ ] 9.2 Establish the client session at `ws://:`, constructed directly from the assignment, ignoring announced hostnames and wss entries and not reusing `ws_entry_address` +- [ ] 9.3 Implement the ingress phase: forward a sphinx packet through the session whose next hop is the agent's own mixnet address, and count arrivals at the shared listener +- [ ] 9.4 Implement the egress phase: send final-hop packets to the gateway's mixnet listener addressed to the agent's own client session, and count arrivals on that session +- [ ] 9.5 Hold the session open across both phases and their drain windows, and use one address family for both legs with the sphinx return hop matching it +- [ ] 9.6 Produce two signals with a fixed two-signal denominator: a phase that produced nothing scores zero, a phase-1 failure does not abort the run, and a session that cannot be established yields two zero signals +- [ ] 9.7 Unit-test the scoring rules of 9.6, including that a healthy-ingress / dead-egress run scores 0.5 rather than 1.0 + +## 10. Orchestrator: per-kind submission + +- [ ] 10.1 Split the result submitter into one stream per kind, each reading and advancing its own watermark and posting to its own endpoint +- [ ] 10.2 Convert a liveness run into its submission shape: the average over the kind's fixed signal set with a missing signal counted as zero, carrying the per-signal breakdown +- [ ] 10.3 Keep the strictly-increasing timestamp behaviour per stream +- [ ] 10.4 Expose the per-signal breakdown and the test kind on the operator read surface (`/v1/results/*`) +- [ ] 10.5 Unit-test that submitting one stream does not advance the other's watermark, and that a failed post leaves its own watermark unmoved + +## 11. nym-api: liveness ingest and shadow-weighted component + +- [ ] 11.1 Add the liveness batch endpoint applying the same ordered validation as the stress endpoint (staleness, contract membership, per-signer monotonicity, signature) +- [ ] 11.2 Scope the per-signer replay high-water mark per endpoint so the stress and liveness streams cannot invalidate each other +- [ ] 11.3 Accept gateway-capable nodes on the liveness endpoint, and store each result with its per-signal breakdown, deduplicating on `(testrun_id, submitter_pubkey)` +- [ ] 11.4 Aggregate liveness results (average performance plus a reachability flag over a window) into a liveness score +- [ ] 11.5 Add the liveness performance component to the provider behind its own `use_*`, `minimum_available_*` and `*_score_weight` flags, with the weight defaulting to ZERO +- [ ] 11.6 Add the divergence metric comparing a node's aggregated liveness score against the v1 routing score, bucketed by whether the node announces a wss entry +- [ ] 11.7 Unit-test that interleaved stress and liveness submissions from one signer are both accepted, that a gateway entry is accepted on the liveness endpoint and dropped on the stress one, and that a zero-weight liveness component leaves detailed performance unchanged + +## 12. Verification + +- [ ] 12.1 `cargo build` the touched workspaces (contracts, nym-node, gateway, nym-api, nym-network-monitor-v3) and confirm no new warnings in the changed crates +- [ ] 12.2 `cargo test` the touched crates, including the contract tests and the orchestrator's sqlx-backed storage tests +- [ ] 12.3 Migrate the contract on a devnet and confirm an existing agent entry still reads back, then confirm a re-announcement populates its identity with no migration logic involved +- [ ] 12.4 Exercise mixnode liveness end to end against a testnet node and confirm a non-zero score with correct per-address attribution +- [ ] 12.5 Exercise gateway liveness end to end against a testnet gateway carrying task groups 6 and 7, and confirm both signals are non-zero +- [ ] 12.6 Confirm an un-upgraded gateway yields a zero egress signal and a non-zero ingress signal, so the divergence bucket behaves as designed +- [ ] 12.7 Confirm the orchestrator migration applies to a copy of a live orchestrator database with its staleness positions, rotation pointers and watermark preserved