Skip to content

fix(evals): validate wire captures and baseline v2.4.0 - #80

Merged
Rynaro merged 1 commit into
mainfrom
fix/65-wire-baseline
Oct 19, 2026
Merged

Rynaro merged 1 commit into
mainfrom
fix/65-wire-baseline

Conversation

@Rynaro

@Rynaro Rynaro commented Oct 19, 2026

Copy link
Copy Markdown
Owner

The archived verifier can report WIRE IDENTICAL for malformed captures, and its v1.11.0 baseline predates the shipped MCP changes. Adds a maintained first-party verifier in Crystalium that validates both raw captures before narrow volatility masking, plus a baseline from the released v2.4.0 source (commit 67a739c). Preserves the historical v1.11.0 capture byte-for-byte.

Capture uses fresh isolated storage and verifies returned content_ref bytes. Provenance records the annotated tag separately from its peeled source commit, source/image hashes, and dependency versions. This is a clean released-source rebuild with freshly resolved dependencies, not a capture of the historical published GHCR image.

Adds repeatable Docker-only make wire-* capture/check/compare commands and 106 regression cases, including paired malformed captures. Independent release captures compare identical. The null-version reproduction returns exit 0 with the old comparator and exit 1 with the new verifier. Current-versus-release comparison records only the exact two reviewed #72 guidance strings; tool names, schemas, defaults and error behavior remain checked.

The nexus comparator remains an archived historical artifact; evals/wire is the documented maintained owner. No files in the external nexus archive are changed.

Validation:

  • make test-ci: 1331 passed, 4 skipped, 1 xfailed.
  • make wire-check: 106 passed; unmounted baked-image tests: 106 passed.
  • Independent v2.4.0 captures: WIRE IDENTICAL.
  • Scoped Ruff/format and whitespace checks pass; independent VIGIL review completed.

Closes #65.
Closes #64 through the maintained replacement and pre-mask shape guards.
Related: #72, #79.

@Rynaro
Rynaro merged commit 0962ceb into main Oct 19, 2026
13 checks passed
@Rynaro
Rynaro deleted the fix/65-wire-baseline branch October 20, 2026 15:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant