Feat/nmv3 liveness checks

openspec/changes/network-monitor-liveness-tests/.openspec.yaml

@@ -0,0 +1,2 @@
+schema: spec-driven
+created: 2026-08-04

openspec/changes/network-monitor-liveness-tests/proposal.md

@@ -0,0 +1,39 @@
+## Why
+
+The nym-api-internal v1 network monitor scores a node's "liveness" by looping packets through a route of five nodes (entry gateway, three mixnodes, exit gateway) and attributing the whole route's delivery ratio to whichever node was substituted in. Its own design record predicted the consequence we now observe in production: "a node that is fine but happens to be tested only against a marginal route can score low", and with `minimum_test_routes = 1` a score "can rest on a single route/gateway path and be noisy". Each node also receives only `per_node_test_packets` (3) packets per route, so a mixnode's reliability is quantised to ninths.
+
+Network monitor v3 already has the machinery to fix this: chain-authorised agents that probe a single node directly and attribute the result to that node alone. Its design record reserves exactly this work, naming "single-hop LIVENESS checks for mixnodes AND gateways, in order to REPLACE the nym-api-internal v1 network monitor" as the known roadmap, and the current implementation already carries the seams (a gateway test type in the data model, gateway-capable nodes retained in the registry, `is_mixnode` / `was_reachable` on every result, and a node-type-agnostic authorisation model). This change builds the first half of that: liveness as a second test kind alongside stress testing, scored per node with no other node in the path.
+
+## What Changes
+
+- **A new `liveness` test kind** alongside the existing stress test, assigned by the orchestrator and executed by the agent. For a mixnode it is the existing two-hop self-loop probe at low volume (order 100 packets rather than 30000). For a gateway it is a new two-phase test.
+- **Gateway liveness is one indivisible test with two phases** over a single client session: an ingress phase (agent as gateway client, packet forwarded out to the agent acting as a mixnode) and an egress phase (agent as mixnode, packet delivered as a final hop back to the agent's client session). Both phases MUST be performed by the same agent in the same run; a phase that produces no signal scores zero rather than being excluded from the average.
+- **Gateway client sessions are established over `ws://<announced-ip>:<clients_ws_port>` only**, ignoring announced hostnames and wss entries. This keeps the tested source IP truthful, removes DNS and certificate handling from the probe, and costs nothing in authenticity because the registration handshake already authenticates the gateway's ed25519 identity. Testing the wss ingress path is explicitly out of scope and recorded as a future test kind.
+- **BREAKING (nym-node):** final-hop packets originating from an authorised network-monitor agent are currently dropped outright ("unsupported network monitor final hop packets"). They MUST instead be processed and delivered to a live client session, and MUST NOT fall back to disk storage. Gateway egress testing is impossible without this, and the no-disk rule keeps monitor traffic from accruing undeliverable rows on every gateway.
+- **BREAKING (nym-node):** a client websocket session whose registration handshake authenticates an ed25519 identity announced on-chain for an authorised network-monitor agent MUST be treated as an ephemeral monitor session: unmetered, and writing nothing to gateway storage. The agent presents no ecash ticketbook.
+- **A contract migration adds an optional ed25519 identity key to each agent entry**, announced by the agent and written by the orchestrator in the existing authorisation transaction. This is what lets the session exemption above key on a cryptographically verified identity rather than on a source IP that k8s host ports share and CNI pools recycle, and it is the gate where an identity is already available: the gateway registration handshake proves possession of the client's ed25519 key before any session exists. The field is additive and optional, which un-upgraded nodes ignore, and it needs no data migration because the contract's agent save is an upsert and agents re-announce before every run.
+- **The agent tests a wave of targets concurrently** rather than one target per invocation, turning the assignment lease bound from the sum of per-target worst cases into the maximum of them, and making a full-network liveness sweep viable at v1's cadence. This requires a shared ingress listener, a multi-target Noise view, and per-target attribution of returned packets.
+- **Per-test-kind scheduling in the orchestrator**: per-kind staleness gates, per-kind address rotation cursors, per-kind lease budgets materialised as an `expires_at` on each in-progress row, and a cooldown that keeps a liveness test from measuring a node still recovering from a stress test. The existing single-in-flight-test-per-node mutex is retained unchanged and now spans kinds.
+- **Per-kind result submission** to nym-api, with a separate watermark per kind and a per-signer replay high-water mark that cannot be shared between kinds.
+- **Liveness enters node performance as a third component with weight zero** (shadow mode) alongside the v1 routing score and the v3 stress score, plus a divergence gauge comparing v3 liveness against v1 routing, bucketed by whether the gateway announces a wss entry so that expected divergence is separable from unexpected divergence.
+
+## Capabilities
+
+### New Capabilities
+
+None. Liveness testing is performed by the same actors, under the same chain authorisation, through the same assignment and submission lifecycle as stress testing, so it belongs in the existing capability rather than duplicating that context.
+
+### Modified Capabilities
+
+- `network-monitors-contract`: adds an optional base58 ed25519 identity key to the agent entry and to `AuthoriseNetworkMonitor`, validated on shape and populated through the existing upsert rather than by a data migration; and records the schema-evolution rule that keeps un-upgraded nodes working, namely additive optional fields only, never a new or retyped `ExecuteMsg` variant.
+- `nym-network-monitor`: adds the liveness test kind and its per-node and per-gateway probe mechanics; amends testrun assignment to be keyed by (node, test kind) with per-kind staleness, rotation and lease budgets; amends the agent lifecycle from one target per invocation to a concurrently-executed wave; amends the node-side gating requirement to permit monitor final-hop delivery without disk fallback and to define the ephemeral unmetered monitor client session; amends the orchestrator storage requirement for the per-kind schema; amends result submission for per-kind watermarks; amends the downstream consumer surface for the shadow-weighted liveness component; and amends the configuration surface for the liveness knobs.
+
+## Impact
+
+- **nym-node** (`src/node/mixnet/handler.rs`): final-hop handling for network-monitor packets. **BREAKING** for gateway egress testing, which cannot work on un-upgraded nodes. Un-upgraded gateways will score zero on the egress phase, which is one reason liveness ships at weight zero.
+- **gateway** (`src/node/client_handling/websocket/connection_handler/{fresh,authenticated}.rs`) and **common/credential-verification** (`bandwidth_storage_manager.rs`): ephemeral, unmetered monitor client sessions. Touches the `ClientDetails` / `BandwidthStorageManager` seam because the storage-assigned `client_id` has no meaning for a session that persists nothing. Also needs a third derived structure alongside the existing routing set and noise map: the announced monitor identities, keyed by ed25519 key rather than by IP, populated from the same startup load and the same nyxd websocket events.
+- **nym-network-monitor-orchestrator**: new migration (per-kind test state, per-kind watermarks, in-progress leases, gateway ws entry address, per-signal result rows), assignment query, node refresher, result submitter, prometheus surface.
+- **nym-network-monitor-agent**: liveness probe profile, wave concurrency with a shared listener and multi-target Noise view, gateway client session and the two-phase gateway probe, ed25519 client identity (derived, not provisioned).
+- **nym-network-monitor-orchestrator-requests**: test-kind-tagged assignments and results carrying per-signal breakdowns, and the announced ed25519 identity on the announce request.
+- **nym-api**: liveness ingest endpoint with its own per-signer replay high-water mark, storage for liveness results, and a shadow-weighted performance component plus the divergence gauge.
+- **contracts/network-monitors** and **common/cosmwasm-smart-contracts/network-monitors-contract**: an optional `bs58_ed25519_identity` on `AuthoriseNetworkMonitor` and on the stored `AuthorisedNetworkMonitor`, validated as base58 decoding to 32 bytes. The contract cannot be redeployed but it can be migrated: it carries a cw2-guarded `migrate` entry point and its admin is the Nymtech SA multisig, so this is a governance action plus a schema bump with no data migration. The field is deliberately additive and optional, because `cw_serde` does not set `deny_unknown_fields` and an un-upgraded node therefore ignores an unknown field on a variant it knows, whereas a new or retyped `ExecuteMsg` variant would fail to parse and make that node silently stop learning about agents. Requires a multisig migration proposal, and no second fleet rollout: the only consumer of the field is a gateway that must already be upgraded for the changes above.

openspec/changes/network-monitor-liveness-tests/specs/network-monitors-contract/spec.md

@@ -0,0 +1,71 @@
+## ADDED Requirements
+
+### Requirement: Schema evolution MUST remain parseable by un-upgraded consumers
+
+The contract's message and response types SHALL only ever be extended in ways that an un-upgraded third-party consumer can still parse. Concretely: a new field on an existing `ExecuteMsg` variant, or on a stored type carried in a query response, is PERMITTED and MUST be optional; introducing a NEW `ExecuteMsg` variant, or changing the type of an existing field, MUST be treated as a breaking fleet change and MUST NOT be used to deliver behaviour that un-upgraded nodes are required to keep observing.
+
+The reason is asymmetric failure. Contract types use `cw_serde`, which does NOT set `deny_unknown_fields`, so a consumer compiled against an older schema silently ignores an unknown field on a variant it recognises and continues to apply the message. A consumer that meets an unrecognised variant, or a field whose type no longer matches, fails deserialisation instead; a Nym node's contract event handler treats that failure as non-fatal, logs that the schema may have changed, and continues processing later blocks. The observable result is not a loud error but a node that has silently stopped learning about agent authorisations and revocations, keeping a stale replay bypass for a revoked address indefinitely.
+
+A change that genuinely requires a new or retyped variant MUST therefore be staged: add the new form alongside the old, wait for the node fleet to carry it, and only then retire the old form.
+
+#### Scenario: An added optional field does not disturb an un-upgraded node
+- **WHEN** an orchestrator sends `AuthoriseNetworkMonitor` carrying a field that a node's compiled schema does not know
+- **THEN** that node parses the message, ignores the unknown field, and still authorises the agent
+
+#### Scenario: A new variant would silently strand an un-upgraded node
+- **WHEN** a hypothetical new `ExecuteMsg` variant is used to authorise or revoke an agent
+- **THEN** an un-upgraded node fails to deserialise it, logs that the schema may have changed, continues with later blocks, and never applies the authorisation or revocation
+
+## MODIFIED Requirements
+
+### Requirement: Only an authorised orchestrator may authorise agents, keyed by socket address as an upsert
+
+`AuthoriseNetworkMonitor { mixnet_address, bs58_x25519_noise, noise_version, bs58_ed25519_identity }` MUST be orchestrator-only, failing with `NotAnOrchestrator` for any other sender. `bs58_x25519_noise` MUST be validated as base58 decoding to exactly 32 bytes (an x25519 noise key), failing with `MalformedX25519AgentNoiseKey` otherwise. On success it MUST save an `AuthorisedNetworkMonitor` keyed by `mixnet_address`, recording `authorised_by = info.sender`, `authorised_at = env.block.time`, and the supplied noise key and version. The save MUST be an upsert: re-authorising the same socket address renews the entry (including `authorised_at`), in contrast to orchestrator authorisation which is a no-op for an existing entry.
+
+`bs58_ed25519_identity` MUST be OPTIONAL, and when present MUST be validated as base58 decoding to exactly 32 bytes (an ed25519 public key), failing with a dedicated malformed-identity error otherwise. It records the ed25519 client identity the agent presents when it opens a gateway client session, which is what allows a gateway to grant an unmetered monitor session against a cryptographically verified identity instead of a source IP. The contract MUST NOT require it, MUST NOT infer it, and MUST NOT treat its absence as an error: an entry without one is a validly authorised agent that simply cannot be recognised on the client-session path. Because the save is an upsert and agents re-announce before every test run, entries written before the field existed acquire it without any data migration or backfill.
+
+The contract places NO uniqueness constraint on `bs58_x25519_noise` OR on `bs58_ed25519_identity`: the same noise key MAY appear under several socket addresses, and does so by design, because a single agent authorises one ipv4 and one ipv6 address so that nodes accept its probes over either family. The registry therefore holds roughly TWO entries per agent, both carrying that agent's noise key and, once announced, the same identity key, and nothing on-chain records that a pair of entries belongs to one agent.
+
+An off-chain consumer that needs to recover which entries belong to one agent MUST group them by that noise key; the two entries of one agent are NOT adjacent in the pagination order, which sorts ipv4 before ipv6. The nym-network-monitor orchestrator does exactly this when it rehydrates its agent cache after a restart. That grouping is only sound as long as distinct agents never share a noise key, and the contract does not enforce it, so this is an assumption held by the consumer rather than an on-chain guarantee. A consumer that builds a set of authorised monitor identities MUST likewise tolerate the same identity arriving from several entries.
+
+#### Scenario: One agent's two addresses are two independent entries
+- **WHEN** an orchestrator authorises one ipv4 and one ipv6 address for the same agent, both with its noise key
+- **THEN** the registry holds two entries sharing that noise key, each independently revocable, with nothing on-chain marking them as one agent
+
+#### Scenario: Only orchestrators can authorise agents
+- **WHEN** an account that is not an authorised orchestrator sends `AuthoriseNetworkMonitor`
+- **THEN** the call fails with `NotAnOrchestrator`
+
+#### Scenario: A malformed noise key is rejected on shape
+- **WHEN** the supplied `bs58_x25519_noise` is not valid base58 or does not decode to exactly 32 bytes
+- **THEN** the call fails with `MalformedX25519AgentNoiseKey`
+
+#### Scenario: Re-authorising the same agent renews the entry
+- **WHEN** an orchestrator authorises an agent for a socket address that already has an entry
+- **THEN** the entry is overwritten with the new `authorised_by`, `authorised_at`, noise key, version, and identity key
+
+#### Scenario: An omitted identity key is accepted
+- **WHEN** an orchestrator authorises an agent without supplying `bs58_ed25519_identity`
+- **THEN** the entry is saved with no identity recorded, and the agent is authorised for every gate that does not depend on one
+
+#### Scenario: A malformed identity key is rejected on shape
+- **WHEN** the supplied `bs58_ed25519_identity` is not valid base58 or does not decode to exactly 32 bytes
+- **THEN** the call fails with a malformed-identity error and nothing is saved
+
+#### Scenario: An entry predating the field acquires it on the next announcement
+- **WHEN** an agent whose stored entry has no identity re-announces and is authorised again
+- **THEN** the upsert records its identity, with no data migration involved
+
+### Requirement: Migration refreshes build information only
+
+`MigrateMsg` MUST be an empty message. `migrate` MUST refresh build information and MUST guard against a downgrade or a wrong contract name via cw2 (`ensure_from_older_version`), and MUST perform no data migration. The `queued_migrations` module MUST contain no migration logic.
+
+This MUST remain true across the addition of the optional agent identity key. That field is deliberately shaped so that no stored entry needs rewriting: an absent value deserialises as `None` under the new schema, and the existing upsert populates it as agents re-announce. A future contract change that cannot be expressed this way MUST add its logic to `queued_migrations` rather than relaxing this requirement silently.
+
+#### Scenario: Migration performs no data rewrite
+- **WHEN** the contract is migrated to a version carrying the optional agent identity key
+- **THEN** build information is refreshed, the cw2 version guard runs, and no agent entry is read or rewritten
+
+#### Scenario: Pre-existing entries remain readable after the migration
+- **WHEN** an agent entry stored before the migration is queried afterwards
+- **THEN** it deserialises with no identity key and every other field unchanged