test: share one SCTP wire harness between the restart runners

diam/peer/testdata/sctprestart/README.md

@@ -1,9 +1,19 @@
 This local SCTP restart harness runs `peer.Manager` against two privileged Linux containers. The alternate peer runs in the primary client's namespace; captures on both `eth0` interfaces cover both associations. No SCTP socket runs on the host. The Go driver is excluded from ordinary package discovery by `testdata` and requires the `sctprestart` build tag.
 
-Run from the repository root, with Docker and the existing `i81-wire-tools:go1.27.2` image available:
+The runner requires Docker with SCTP support, Bash and Python 3. It uses
+`scripts/sctp-wire/harness.sh`, the shared Dockerfile and tshark exporter.
+Every runner independently builds `sctp-wire-tools:go1.27.2` when absent or
+verifies it when present, then records and uses its immutable image ID. No sm
+run is required first. Set `SCTP_WIRE_IMAGE` to select a different or fresh tag.
+Python 3 runs the package-local analyzer; tshark runs in the recorded image.
+
+Invoke the runner from any directory using its path. Arguments remain
+`[source checkout] [output root] [variant]`, with this checkout, a unique
+`sctp-wire-peer-evidence-*` directory under `${TMPDIR:-/tmp}`, and `basic`
+as defaults. For example, from the repository root:
 
 ```sh
-diam/peer/testdata/sctprestart/run.sh "$PWD" /tmp/i93-wire basic
+diam/peer/testdata/sctprestart/run.sh "$PWD" /path/to/evidence/peer basic
 ```
 
 The third argument selects `basic`, `changed-identity`, `changed-apps`, `alternate-stream`, `fragment`, `watchdog`, or `deadline`. Each run writes evidence under a subdirectory named for that variant. Pass a frozen source checkout as the first argument to compare baseline and changed implementations; the builder overlays only this driver onto that checkout. The driver compiles against the public Manager APIs on both versions.
@@ -18,17 +28,22 @@ Watchdog analysis uses the replacement CER's host and the restart hook's new gen
 
 The harness wrapper forwards each message synchronously to Manager. After `ServeDIAM` returns from protocol commit, it reads `Manager.Peers()` and records a `RECOVERY_CHECKPOINT` containing the processed CER/DWA command, both identifiers, connection generation and peer snapshot. A TCP control connection on server port 3869 delivers that checkpoint to the client. The client waits for the CER checkpoint before answering recovery probes, and for each DWA checkpoint before sending the next DWA. The analyzer requires REOPEN/ineligible at admission and after DWA 1 and 2, then OKAY/eligible after DWA 3 (RFC 3539 section 3.4.1 and Appendix A). Every checkpoint must match the corresponding captured message and precede the next answer's delivery. Asynchronous `OnPeerEvent` callbacks do not establish recovery boundaries; delayed, missing or queued recovery callbacks cannot affect this check. Initial admission and the alternate peer cannot satisfy recovery. No additional Manager accessor is required.
 
-Run the analyzer's synthetic packet/log regression suite in Docker:
+Run the analyzer's synthetic packet/log regression suite from the repository root:
 
 ```sh
-docker run --rm --user "$(id -u):$(id -g)" \
-  --mount "type=bind,src=$PWD,dst=/source,readonly" \
-  -w /source/diam/peer/testdata/sctprestart \
-  i81-wire-tools:go1.27.2 python3 -B -m unittest -v analyze_test
+python3 -B -m unittest discover -s diam/peer/testdata/sctprestart -p '*_test.py' -v
 ```
 
 These tests execute the analyzer CLI with synthetic tshark output. They check its acceptance and rejection boundaries; the live two-container watchdog run supplies packet capture evidence.
 
-Missing Docker, the image, SCTP support, tshark, captures, restart evidence, or assertions fails the command. Evidence includes packet tables, raw application request bytes, captures and SHA-256 hashes, logs, build and dependency versions, image identity, kernel version, and receiver drop counters. Run-specific containers and networks are labeled, removed on every exit path, and checked by label and name. No volumes are created.
+Missing Docker, SCTP support, tshark, captures, restart evidence, or assertions fails the command. Evidence includes packet tables, raw application request bytes, captures and SHA-256 hashes, logs, build and dependency versions, image identity, kernel version, and receiver drop counters. Run-specific containers and networks are labeled, removed on every exit path, and checked by label and name. No volumes are created.
+
+Compilation copies source into the builder's writable
+container layer and mounts no host Go caches. Artifacts are copied out as the
+calling user; privileged peers bind only the binary. Container/network names
+use unique `sctp-wire-*` prefixes and temporary directories use
+`sctp-wire-build.*`. Docker CLI configuration stays in that temporary
+directory. Set `TMPDIR` to choose its location; no files are written outside
+that directory and the output directory.
 
 The harness does not test TLS, dialed association renewal, packet loss/reordering, or peer state election races. Unit and scripted transport tests cover those policy and concurrency cases. Its packet checks and observed generations establish a real accepted SCTP association restart, not a fresh listener connection.

diam/peer/testdata/sctprestart/analyze.py

@@ -3,7 +3,7 @@
 import json
 import pathlib
 import re
-import subprocess
+import runpy
 import sys
 
 
@@ -92,18 +92,16 @@ def assert_watchdog_recovery(rows, log, cer, cea):
 
 
 out = pathlib.Path(sys.argv[1]).resolve()
-image = 'i81-wire-tools:go1.27.2'
-run_id = (out / 'run-id').read_text().strip()
+tshark = runpy.run_path(str(pathlib.Path(__file__).resolve().parents[4] / 'scripts/sctp-wire/tshark.py'))['tshark']
 fields = ['frame.number', 'frame.time_epoch', 'ip.src', 'sctp.chunk_type', 'sctp.srcport', 'sctp.dstport', 'diameter.flags.T', 'diameter.Session-Id', 'diameter.User-Name', 'sctp.data_sid', 'diameter.cmd.code', 'diameter.flags.request', 'diameter.hopbyhopid', 'diameter.endtoendid', 'diameter.Result-Code', 'diameter.Origin-Host', 'diameter.Auth-Application-Id', 'diameter.Origin-State-Id']
-cmd = ['docker', 'run', '--rm', '--label', f'harness={run_id}', '--mount', f'type=bind,src={out},dst=/evidence,readonly', image, 'tshark', '-r', '/evidence/client.pcapng', '-Y', 'sctp', '-T', 'fields']
+arguments = ['-Y', 'sctp', '-T', 'fields']
 for field in fields:
-    cmd.extend(['-e', field])
-result = subprocess.run(cmd, text=True, capture_output=True, check=True)
+    arguments.extend(['-e', field])
+result = tshark(out, 'client.pcapng', *arguments)
 (out / 'packets.tsv').write_text(result.stdout)
 (out / 'tshark-errors.txt').write_text(result.stderr)
 rows = [dict(zip(fields, row)) for row in csv.reader(result.stdout.splitlines(), delimiter='\t')]
-server_cmd = [arg.replace('/evidence/client.pcapng', '/evidence/server.pcapng') for arg in cmd]
-server_result = subprocess.run(server_cmd, text=True, capture_output=True, check=True)
+server_result = tshark(out, 'server.pcapng', *arguments)
 (out / 'server-packets.tsv').write_text(server_result.stdout)
 server_rows = [dict(zip(fields, row)) for row in csv.reader(server_result.stdout.splitlines(), delimiter='\t')]
 assert any(row['sctp.chunk_type'] == '6' for row in server_rows), 'ABORT did not reach receiver capture point'
@@ -167,8 +165,7 @@ def assert_watchdog_recovery(rows, log, cer, cea):
 assert original['diameter.hopbyhopid'] != retry['diameter.hopbyhopid'], 'hop-by-hop not freshly allocated'
 for key in ('diameter.Session-Id', 'diameter.User-Name'):
     assert original[key] == retry[key] and original[key], f'{key} payload changed or absent'
-raw_cmd = cmd[:cmd.index('-Y')] + ['-Y', 'diameter.cmd.code == 272 && diameter.flags.request == 1', '-T', 'jsonraw', '-j', 'diameter']
-raw = subprocess.run(raw_cmd, text=True, capture_output=True, check=True)
+raw = tshark(out, 'client.pcapng', '-Y', 'diameter.cmd.code == 272 && diameter.flags.request == 1', '-T', 'jsonraw', '-j', 'diameter')
 (out / 'request-raw.json').write_text(raw.stdout)
 wire = [bytes.fromhex(p['_source']['layers']['diameter_raw'][0]) for p in json.loads(raw.stdout)]
 assert len(wire) == 2 and wire[0][20:] == wire[1][20:], 'complete serialized AVP payload changed on retry'

diam/peer/testdata/sctprestart/run.sh

@@ -1,80 +1,28 @@
 #!/usr/bin/env bash
-# Every SCTP socket and packet capture runs in isolated privileged containers.
+# Scenario commands and timings; shared lifecycle lives in scripts/sctp-wire.
 set -euo pipefail
 harness=$(cd "$(dirname "$0")" && pwd)
-source=$(cd "${1:-$harness/../../../..}" && pwd)
+root=$(cd "$harness/../../../.." && pwd)
+# shellcheck source=scripts/sctp-wire/harness.sh
+source "$root/scripts/sctp-wire/harness.sh"
+scenario=peer
 variant=${3:-basic}
 case "$variant" in basic|changed-identity|changed-apps|alternate-stream|watchdog|fragment|deadline) ;; *) echo "unknown variant: $variant" >&2; exit 2 ;; esac
-rig=$(mktemp -d "${TMPDIR:-/tmp}/i93-wire-build.XXXXXX")
-trap 'rm -rf "$rig"' EXIT
-name=retained
+source=$(cd "${1:-$harness/../../../..}" && pwd)
+output=${2:-${TMPDIR:-/tmp}/sctp-wire-peer-evidence-$(date +%s)-$$}
+output=$output/$variant
+fixture=diam/peer/testdata/sctprestart/main.go
+build=(build -tags=sctprestart -o /out/peer ./diam/peer/testdata/sctprestart)
 binary=peer
-out=$(realpath -m "${2:-/tmp/i93-wire-evidence-$(date +%s)-$$}/$variant")
-mkdir -p "$out"
-run=i93-wire-${name}-${variant}-${rig##*.}
-network=$run-net
-server=$run-server
-client=$run-client
-builder=$run-build
-analyzer=$run-analyze
-
-cleanup(){
- for peer in "$client" "$server"; do
-  for child in capture initial server alternate; do
-   signal=TERM
-   if [[ $child == capture ]]; then signal=INT; fi
-   if [[ $child == initial ]]; then signal=KILL; fi
-   if docker exec "$peer" test -s "/tmp/$child.pid" >/dev/null 2>&1; then
-    if ! docker exec "$peer" test -s "/tmp/$child.exit" >/dev/null 2>&1; then
-     docker exec "$peer" sh -c 'kill -"$1" "$(cat "/tmp/$2.pid")"' sh "$signal" "$child" >/dev/null 2>&1 || true
-     for ((i=0;i<10;i++)); do
-      if docker exec "$peer" test -s "/tmp/$child.exit" >/dev/null 2>&1; then break; fi
-      sleep .1
-     done
-    fi
-   fi
-  done
-  role=${peer##*-}
-  for file in wire.pcapng capture.log server.log initial.log alternate.log; do
-   destination=$out/$role-$file
-   if [[ ! -e $destination ]]; then docker cp "$peer:/tmp/$file" "$destination" >/dev/null 2>&1 || true; fi
-  done
- done
- docker rm -f -v "$client" "$server" "$builder" "$analyzer" >/dev/null 2>&1 || true
- docker network rm "$network" >/dev/null 2>&1 || true
- docker ps -a --filter "label=harness=$run" --format '{{.Names}}' > "$out/cleanup-containers.txt"
- docker volume ls --filter "label=harness=$run" --format '{{.Name}}' > "$out/cleanup-volumes.txt"
- docker network ls --filter "label=harness=$run" --format '{{.Name}}' > "$out/cleanup-networks.txt"
- docker ps -a --filter "name=$run" --format '{{.Names}}' > "$out/cleanup-name.txt"
- docker volume ls --filter "name=$run" --format '{{.Name}}' > "$out/cleanup-name-volumes.txt"
- docker network ls --filter "name=$run" --format '{{.Name}}' > "$out/cleanup-name-networks.txt"
- rm -rf "$rig"
-}
-trap cleanup EXIT
-trap 'exit 130' INT TERM
-printf '%s\n' "$run" > "$out/run-id"
-cp "$harness/Dockerfile" "$rig/Dockerfile"
-docker image inspect i81-wire-tools:go1.27.2 > "$out/image-build.log"
-mkdir -p "$rig/out"
-docker run --rm --name "$builder" --label "harness=$run" \
- --mount "type=bind,src=$source,dst=/source,readonly" \
- --mount "type=bind,src=$harness,dst=/harness,readonly" \
- --mount "type=bind,src=$rig/out,dst=/out" \
- i81-wire-tools:go1.27.2 sh -c 'cp -a /source /tmp/build; mkdir -p /tmp/build/diam/peer/testdata/sctprestart; cp /harness/main.go /tmp/build/diam/peer/testdata/sctprestart/main.go; cd /tmp/build; for f in diam/server.go diam/accepted_generation.go diam/network_sctp.go diam/peer/manager.go diam/peer/actor.go diam/peer/session.go diam/peer/renewal.go diam/peer/watchdog.go diam/peer/send.go diam/peer/testdata/sctprestart/main.go go.mod go.sum; do if test -f "$f"; then sha256sum "$f"; fi; done > /out/source-hashes.txt; go build -tags=sctprestart -o /out/peer ./diam/peer/testdata/sctprestart' > "$out/build.log" 2>&1
-
-cp "$rig/out/source-hashes.txt" "$out/source-hashes.txt"
+wire_path=/wire
+server_port=3868
+client_port=43868
+capture_suffix=
+cleanup_attempts=10
+analyze_on_failure=true
+hashes=(diam/server.go diam/accepted_generation.go diam/network_sctp.go diam/peer/manager.go diam/peer/actor.go diam/peer/session.go diam/peer/renewal.go diam/peer/watchdog.go diam/peer/send.go diam/peer/testdata/sctprestart/main.go go.mod go.sum)
 
-docker network create --label "harness=$run" "$network" >/dev/null
-for peer in "$server" "$client"; do
- docker run -d --privileged --label "harness=$run" --name "$peer" --network "$network" --mount "type=bind,src=$rig/out/$binary,dst=/wire,readonly" i81-wire-tools:go1.27.2 sleep infinity >/dev/null
-done
-sip=$(docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' "$server")
-cip=$(docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' "$client")
-docker exec "$server" iptables -I INPUT -p sctp -s "$cip" --sport 43868 --dport 3868 -m sctp --chunk-types any ABORT -j DROP
-for peer in "$server" "$client"; do
- docker exec -d "$peer" sh -c 'tshark -i eth0 -f sctp -w /tmp/wire.pcapng >/tmp/capture.log 2>&1 & capture_pid=$!; printf "%s\n" "$capture_pid" >/tmp/capture.pid; wait "$capture_pid"; printf "%s\n" "$?" >/tmp/capture.exit'
-done
-sleep 1
+wire_scenario() {
 watchdog=false
 if [[ $variant == watchdog ]]; then watchdog=true; fi
 request_timeout=12s
@@ -117,42 +65,6 @@ result=$?
 set -e
 printf '%s\n' "$result" > "$out/restart.exit"
 if [[ $variant == deadline ]]; then sleep 6; else sleep .5; fi
-for peer in "$server" "$client"; do
- docker exec "$peer" sh -c 'kill -INT "$(cat /tmp/capture.pid)"'
- for ((i=0;i<30;i++)); do if docker exec "$peer" test -s /tmp/capture.exit; then break; fi; sleep .1; done
- docker exec "$peer" test -s /tmp/capture.exit
- docker cp "$peer:/tmp/wire.pcapng" "$out/${peer##*-}.pcapng" >/dev/null
- docker cp "$peer:/tmp/capture.log" "$out/${peer##*-}-capture.log" >/dev/null
- docker cp "$peer:/tmp/capture.exit" "$out/${peer##*-}-capture.exit" >/dev/null
-done
-docker cp "$server:/tmp/server.log" "$out/server.log" >/dev/null
-docker cp "$client:/tmp/initial.log" "$out/initial.log" >/dev/null
-docker cp "$client:/tmp/alternate.log" "$out/alternate.log" >/dev/null
-docker cp "$client:/tmp/initial.exit" "$out/initial.exit" >/dev/null
-docker exec "$server" iptables -nvxL INPUT > "$out/iptables.txt"
-docker exec "$server" uname -a > "$out/kernel.txt"
-docker exec "$server" /usr/local/go/bin/go version > "$out/go-version.txt"
-docker exec "$server" /usr/local/go/bin/go version -m /wire > "$out/binary-build.txt"
-docker inspect "$server" --format '{{.Image}}' > "$out/image-id.txt"
-docker exec "$server" tshark --version > "$out/tshark-version.txt"
-docker exec "$server" sh -c 'kill -TERM "$(cat /tmp/server.pid)"'
-for ((i=0;i<30;i++)); do if docker exec "$server" test -s /tmp/server.exit; then break; fi; sleep .1; done
-docker exec "$server" test -s /tmp/server.exit
-printf '%s\n' "run=$run restart_exit=$result output=$out"
-cat "$out/restarted.log"
-cleanup
-socket_gid=$(docker run --rm --label "harness=$run" --user "$(id -u):$(id -g)" \
- --mount type=bind,src=/var/run/docker.sock,dst=/var/run/docker.sock \
- i81-wire-tools:go1.27.2 stat -c '%g' /var/run/docker.sock)
-docker run --rm --name "$analyzer" --label "harness=$run" \
- --user "$(id -u):$(id -g)" --group-add "$socket_gid" \
- --mount type=bind,src=/var/run/docker.sock,dst=/var/run/docker.sock \
- --mount type=bind,src=/usr/bin/docker,dst=/usr/bin/docker,readonly \
- --mount "type=bind,src=$harness/analyze.py,dst=/analyze.py,readonly" \
- --mount "type=bind,src=$out,dst=$out" \
- i81-wire-tools:go1.27.2 python3 -B /analyze.py "$out" > "$out/assertions.txt" 2>&1 || result=$?
-cat "$out/assertions.txt"
-sha256sum "$out"/*.pcapng > "$out/capture-sha256.txt"
-cleanup
-trap - EXIT
-exit "$result"
+}
+
+wire_run