Fair measurements instead of client-only

.github/workflows/lint.yaml

@@ -62,6 +62,7 @@ jobs:
           cd typhoon && cargo fmt --check
           cd ../evaluation/protocols/typhoon && cargo fmt --check
           cd ../wireguard_daita/machines-printer && cargo fmt --check
+          cd ../../quic/quic-eval && cargo fmt --check
 
       - name: Run clippy 📎
         working-directory: typhoon
@@ -104,12 +105,15 @@ jobs:
       - name: Setup Go 🦫
         uses: actions/setup-go@v5
         with:
-          go-version: "1.22"
+          go-version: "1.23"
+
+      - name: Install staticcheck 🔧
+        run: go install honnef.co/go/tools/cmd/staticcheck@latest
 
       - name: Lint Go (wireguard_daita) 🔍
         working-directory: evaluation/protocols/wireguard_daita/wg-daita
         run: |
           git clone --depth=1 --branch mullvad https://github.com/mullvad/wireguard-go ../wireguard-go
-          go install honnef.co/go/tools/cmd/staticcheck@latest
           go mod tidy
+          go vet -tags daita ./...
           staticcheck -tags daita ./...

.gitignore

@@ -14,6 +14,12 @@ typhoon/.test_keys
 
 # Evaluation:
 
+evaluation/transport/target
+evaluation/transport/Cargo.lock
+
+evaluation/protocols/quic/quic-eval/target
+evaluation/protocols/quic/quic-eval/Cargo.lock
+
 evaluation/protocols/wireguard_daita/wg-daita/go.sum
 evaluation/protocols/wireguard_daita/wireguard-go
 

PROTOCOL.md

@@ -1472,6 +1472,27 @@ Flushing too eagerly (on every packet) eliminates the syscall reduction; flushin
 The queue must also account for the fact that `sendmmsg` batches datagrams for a single socket, while the server may manage multiple sockets simultaneously (one per flow on SO_REUSEPORT platforms), meaning coalescing must happen per-socket.
 The right flush policy — fixed threshold, fixed deadline, or adaptive — requires empirical benchmarking under realistic load patterns.
 
+#### Multi-job send/receive queue routine
+
+The batching described above coalesces the wire packets of a _single_ user message, but the dominant cost under sustained bulk transfer is different: profiling a saturated flow shows the process spending the large majority of its on-CPU time inside the per-datagram `sendto`/`recvfrom` syscalls, with symmetric crypto a rounding error by comparison.
+The reason is architectural rather than cryptographic — the current data path issues one syscall per wire packet and, on the sender side, the caller `await`s each `send_bytes` through the full flow-manager stack before the next packet is prepared, so throughput is bounded by _per-packet round-trips through the executor_ rather than by how fast bytes can be encrypted.
+A multi-job queue routine decouples the application-visible send/receive calls from the syscalls that move bytes, so that many packets amortise into few kernel crossings and the crypto work spreads across cores.
+
+**Send path.**
+Each socket owns a bounded multi-producer queue of ready-to-send jobs.
+A `send_bytes` call splits its user message into wire packets, performs the trailer/crypto work — optionally handed to a small crypto worker pool so several messages encrypt in parallel — and enqueues the finished `(wire_bytes, addr)` jobs, returning as soon as they are accepted rather than after the datagrams have left the host.
+A dedicated per-socket flusher job drains the queue and issues one `sendmmsg` per drained batch, bounded by a maximum batch width (the `sendmmsg` vector limit) and a flush policy that fires on whichever of a fill threshold or a short deadline is reached first.
+Decoy emission feeds the same queue, so real and decoy datagrams interleave in the batched syscall exactly as they would on the wire and no separate send path can be timed apart from the data path.
+
+**Receive path.**
+A dedicated per-socket reader job pulls a batch of datagrams in a single `recvmmsg` call and hands the raw buffers to a decrypt worker pool; each worker locates and verifies the trailer, strips fake header/body padding, and routes the recovered payload to the owning session's receive queue keyed by identity.
+Batched receive matters most on the server, where a single listening socket may carry datagrams for many concurrent sessions in one `recvmmsg` sweep; per-session ordering is preserved because a single reader assigns arrival order before fan-out, and the worker pool only parallelises the per-packet crypto, not the enqueue.
+
+**The challenge**: the queue routine reintroduces, at a larger granularity, every tension the single-message batching already faces — flush eagerness versus syscall amortisation, and per-socket rather than per-process coalescing — but adds three of its own.
+First, ordering: parallel crypto workers may finish out of order, so the design must either restore per-session sequence before delivery or rely on the existing trailer sequencing to tolerate reordering, and must not let a slow worker head-of-line-block an unrelated session.
+Second, backpressure: when a queue fills — a stalled link, a slow consumer — `send_bytes` must block or shed rather than grow memory unboundedly, and that backpressure has to surface to the caller without deadlocking the flusher that drains the same queue.
+Third, timing fidelity: coalescing must not distort the very inter-packet timing the flow shaper works to produce, so the flush deadline has to stay well below the smallest inter-arrival gap the shaper and health-check scheduler rely on, or the batching becomes a fingerprint of its own.
+
 ### Per-deployment randomisation seed
 
 In the current design, every TYPHOON deployment draws `FlowConfig` and decoy parameters from the same global distributions defined in `TYPHOON_*` settings keys.

README.md

@@ -246,7 +246,7 @@ At default sizes, both TYPHOON _and_ other UDP-native services (WireGuard, OpenV
 The [`evaluation/`](evaluation/) directory contains a Docker-based traffic capture and analysis harness, organised into three independent parts:
 
 1. **TYPHOON self-comparison** — measure run-to-run and scenario-to-scenario variability of TYPHOON's own traffic profile.
-2. **Operational comparison** — capture all 16 protocols (TYPHOON + 15 comparators) under a controlled Docker network and compare throughput, overhead, goodput efficiency, byte entropy, burstiness, and handshake metrics. _Operational, not detectability._
+2. **Operational comparison** — run a per-packet latency ping against all 16 protocols (TYPHOON + 15 comparators) under a controlled Docker network and compare **delivery** and the **round-trip latency distribution** (clean and under netem loss/jitter), plus wire-shape metrics (overhead, entropy, burstiness, handshake). _Operational, not detectability._
 3. **Background-blending evaluation** — generate a corpus of natural UDP traffic (QUIC HTTPS, DNS, RTP voice/video, gaming, control plane), run TYPHOON alongside, and measure how often a passive classifier mistakes TYPHOON for benign traffic.
 
 See [evaluation/README.md](evaluation/README.md) for requirements, install steps, CLI reference, and instructions for reading the results.

evaluation/README.md

@@ -8,7 +8,8 @@ For the protocol itself, see [PROTOCOL.md](../PROTOCOL.md).
 
 ```text
 evaluation/
-├── protocols/             # Dockerfiles for the 15 comparison protocols
+├── protocols/             # Dockerfiles for the 16 comparison protocols
+├── transport/             # shared Rust crate: the senders/sinks every protocol runs
 ├── background/            # Dockerfiles for the 8 UDP-traffic generators + open-set unknown
 ├── compose/, observer/    # docker-compose stacks and the tcpdump observer container
 ├── chaos/                 # tc/netem sidecar for latency / jitter / loss
@@ -30,7 +31,11 @@ Four independent parts; each answers one question.
 
 ### Part 2 — Operational comparison
 
-*Where does TYPHOON sit on throughput / overhead / handshake-cost relative to 15 other UDP/TCP secure-transport protocols?* Captures each protocol once under the same Docker network and emits direct operational metrics. **No classifiers** — a closed-world classifier across these 16 protocols would score near 100 % because each protocol has a distinct wire footprint by design, which says nothing about Part 3's question.
+*How does TYPHOON compare on reliability and cost to 15 other UDP/TCP secure-transport protocols?*
+Runs a **per-packet latency ping** against each: a spaced sequence of small equal-sized probes (500 × 256 B @ 20 ms) that the sink echoes, measuring per-packet **round-trip time** in tunnel-like conditions.
+It reports **delivery** (what fraction the receiver gets / echoes back) and the **RTT distribution**.
+See [Operational metrics](#operational-metrics-delivery--latency) for what actually discriminates the protocols (spoiler: the tail, not the median).
+**No classifiers** — a closed-world classifier across these 16 protocols would score near 100 % because each protocol has a distinct wire footprint by design, which says nothing about Part 3's question.
 
 Protocols compared: `raw_udp`, `raw_tcp`, `tls`, `wireguard`, `quic`, `obfs4` (×3 IAT modes), `amneziawg`, `hysteria2`, `shadowsocks`, `tor`, `vless_reality`, `openvpn`, `wireguard_daita`, `typhoon`.
 
@@ -88,13 +93,13 @@ poe plot --example heavy_traffic --out-dir out/   # per-flow packet-structure SV
 ### Part 2 — Operational comparison (CLI)
 
 ```shell
-poe capture --all             # capture all 16 protocols (default: bulk, 10 MB)
-poe capture --all --chaos     # …with latency + jitter + loss
-poe analyze                   # parse pcaps → stats.json
+poe capture --all             # per-packet latency ping across all 16 protocols
+poe capture --all --chaos     # …under netem (2% loss, 100 ms±30 ms delay)
+poe analyze                   # parse pcaps → stats.json (detectability metrics)
 poe proto-compare             # plots + comparison table for the latest run
 ```
 
-Useful flags: `--protocol <name>` (single protocol), `--scenario {bulk,interactive,streaming,burst,echo,idle}`, `--run YYYYMMDD_HHMMSS` (target an earlier run).
+Useful `capture` flags: `--protocol <name>` (single protocol); `--chaos` + `--loss-pct <n>`; `--seed <n>` (reproducible). `--profile <name>` selects the TYPHOON *settings* profile (fake-body mode, decoys); `bulk_upload` is the operational default, the rest are the Part-3 mimicry profiles. `analyze`/`proto-compare` take `--run YYYYMMDD_HHMMSS` to target an earlier run.
 
 ### Part 3 — Background-blending (CLI)
 
@@ -155,8 +160,8 @@ Outputs are split into two trees:
 results/
 ├── captures/run_<timestamp>/
 │   ├── <protocol>.pcap            # raw capture (handshake + data)
-│   ├── stats.json                 # per-pcap metrics — see below
-│   ├── metadata.json              # transfer_bytes, scenario, timing
+│   ├── stats.json                 # per-pcap detectability metrics — see below
+│   ├── metadata.json              # delivery %, per-packet RTT distribution (rtt_*)
 │   └── logs/<protocol>/           # client + server + observer container logs
 └── background/pipeline_<id>/run_*/  # Part 3 per-run pcaps + metadata
 
@@ -175,6 +180,28 @@ artifacts/pipeline_<timestamp>/
     └── distplot/                  # per-pair size/IAT overlays PDFs + JSON
 ```
 
+### Operational metrics (delivery / latency)
+
+Part 2's numbers come from the two endpoints (recorded in `metadata.json`), not the pcap:
+
+| Metric | Meaning |
+| --- | --- |
+| `delivery_pct` | probes the sink received ÷ sent (one-way); `roundtrip_delivery_pct` is the client's echoes-received ÷ sent |
+| `rtt_min / p50 / p95 / p99_ms`, `rtt_jitter_ms` | per-packet round-trip distribution |
+
+**Read the tail, not the median.**
+The p50 RTT is dominated by the shared path — the observer hop in clean mode, the netem delay under chaos — and is ~identical across protocols, so it is *not* a discriminator.
+The signal lives in two places:
+
+- **Delivery** separates best-effort from reliable: under chaos, UDP-family transports settle at the raw ~2 % one-way loss (they *drop* it) while TCP/QUIC/proxies stay 100 % (they *retransmit*).
+- **The latency tail (p95/p99)** exposes the *cost* of a protocol's design that the median hides:
+  - clean — traffic-shapers pay a visible tail (wireguard_daita p95 ~45 ms, obfs4 IAT-mode ~24 ms) from their padding / inter-arrival machines, while plain transports sit at ~7 ms;
+  - chaos — reliable transports pay a retransmit spike (p99 ~600–850 ms) that best-effort UDP does not (p99 ≈ p95).
+
+So the metric answers *"how reliable is it, and what does its obfuscation / loss-recovery cost in latency"* — e.g. TYPHOON delivers like the other UDP transports yet keeps a flat tail (its decoys / fake-headers add no measurable latency penalty), unlike DAITA.
+It is deliberately **not** a throughput benchmark: the shared capture point and best-effort UDP's lack of congestion control make bulk throughput an artifact *across* transport classes (an earlier bulk-transfer design was abandoned for exactly this reason — see the git history).
+For the TYPHOON implementation's raw speed on loopback, free of these network artifacts, see **Part 4** (`cargo bench`).
+
 ### Per-pcap metrics (`stats.json`)
 
 Computed separately per direction (`c2s`, `s2c`, `all`). Packet sizes are **transport-payload bytes** (UDP payload or TCP segment data) — IP/UDP/TCP header bytes are excluded so transport overhead doesn't leak into protocol stats.
@@ -201,7 +228,7 @@ Computed separately per direction (`c2s`, `s2c`, `all`). Packet sizes are **tran
 
 ### Part 2 plots (under `artifacts/<pipeline_id>/proto_compare/`)
 
-- `run_<id>_proto_compare.pdf` — six panels: (A) size CDF, (B) IAT CDF, (C) throughput vs goodput-efficiency scatter, (D) overhead bars, (E) byte entropy by phase, (F) normalised heatmap.
+- `run_<id>_proto_compare.pdf` — six panels: (A) size CDF, (B) IAT CDF, (C) per-packet RTT vs delivery scatter, (D) overhead bars, (E) byte entropy by phase, (F) normalised heatmap.
 - `run_<id>_handshake.pdf` — handshake duration / packet count / byte fraction across protocols.
 - `run_<id>_compare_table.md` — one row per protocol; quick-glance ranking by any column.