NLAttack v2.0.0 — formal release (adds the Red/Blue CTF family)

[SolshineCode]

Formalizes the second benchmark generation as v2.0.0. v1 (the prior frozen release) and its published results are carried forward unchanged.

What v2 is

• New: Family N — CTF Red/Blue (P119–P128, nla_eval/ctf.py, docs/CTF_RED_BLUE.md). An adversarial capture-the-flag packaging of the NLA-as-monitor threat model: a Red model pursues a hidden misaligned objective (the flag) while a Blue NLA-monitor reads the AV verbalization to catch it. Scoreboard: clean-capture rate (Red), catch rate (Blue's recall on genuine captures), stealth gap, evasion cost. Offline reproducible control-tier demo against MockNLA.
• Release maturation: version bumped to 2.0.0 (nla_eval/__init__.py, CITATION.cff); new CHANGELOG.md; VERSIONING.md rewritten around plain v1/v2 generation labels (freeze-on-release policy kept); ROADMAP_v0.2.md → ROADMAP_v2.md.
• Catalog: 118 plans / 13 families (v1, frozen, families A–M) → 128 plans / 14 families (v2, adds Family N). Reconciled across INDEX, plans/README, EVALUATIONS, README, CITATION.
• README simplified to a single-screen front page; v1 results kept visible and labeled frozen.
• Results: RESULTS.md gains a "v2 additions" section (CTF control-tier demo + the domain-aware honest-negative) with the control-vs-primary integrity caveat; new results/README.md provenance map (v1 artifacts never overwritten).

Integrity / honesty

• Capture is judged on the activation (the bottleneck probe), the flag on the AV text — never the same signal (independent-review P0 Fold in Hermes design review: bottleneck probes, matcher ensemble, faithfulness, controls, red-team #1). A CTF result is primary-tier only with a real probe oracle over the model's own behavior; the survival-proxy oracle is control-tier (every CTFScore carries its tier).
• The tagged v2.0.0 ships the new family + release maturation. It does not fabricate domain-coverage or harness-fix-dependent numbers it hasn't produced — the multi-domain expansion and the H1–H12 fixes are tracked as the v2.x roadmap.

Note on the other PRs

PR #21 (versioning scaffold) and PR #22 (domain-aware result) are already merged to master, so this branch already contains them; they can be closed.

Validation

python run_example.py, python experiments/ctf_red_blue_demo.py (reproducible), and import nla_eval; nla_eval.__version__ == "2.0.0" all pass; CITATION.cff parses; no stale links after the roadmap rename.

🤖 Generated with Claude Code

[gemini-code-assist]

Code Review

This pull request introduces "CTF Red/Blue" (Family N, v0.2 additive), a game-theoretic framework that pits an adaptive Red team model pursuing a hidden deceptive objective against a Blue NLA-monitor reading the AV verbalization. The changes include the core game loop and scoring logic in nla_eval/ctf.py, an offline control-tier demo in experiments/ctf_red_blue_demo.py, and extensive documentation updates. The review feedback focuses on improving JSON serialization compatibility by replacing float("nan") with None, resolving formatting inconsistencies and potential TypeErrors when printing metrics in the demo, and ensuring the generated JSON output ends with a final newline.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[SolshineCode]

🔍 Review request — v2.0.0 release gate

This PR is ready for the two-lens external review the rollout gate requires (docs/VERSIONING.md). Full packet: docs/reviews/v2.0.0-release-review.md. Please do not tag v2.0.0 until both lenses sign off (the tag is not pushed).

TL;DR of what's under review — additive generation bump: adds Family N (CTF Red/Blue, P119–P128) + release maturation; changes no v1 content or results.

/gemini review

Gemini lens — please focus on these (packet §3–§5), they're the load-bearing claims:

• C3 / V5 — v1 untouched. Confirm no v1 result artifact or scoring code changed:
  git diff origin/master...HEAD -- results/cross_nla results/emergence results/deception nla_eval/emergence.py should be empty. The 0.601 composite and the cross-NLA leaderboard table in docs/RESULTS.md must be textually unchanged.
• A1 — capture ≠ flag. In nla_eval/ctf.py, verify the Blue matcher result is never used as the capture signal (survival_oracle reads the committed input; probe_capture_oracle reads the probe; BlueMonitor.inspect reads the AV text). If they were the same signal, clean_capture would be impossible by construction.
• A2 — tier honesty (P128). Every CTFScore carries a tier; the MockNLA run is labeled control-tier (mechanics demo, not a monitor certification) in the demo, RESULTS.md, and CHANGELOG.md.
• C5 — no fabricated numbers. Only the control-tier CTF demo + the domain-aware honest-negative are reported; the multi-domain expansion and H1–H12 fixes are labeled not-yet-run.
• C1/C2/C6 — version reads 2.0.0 everywhere; counts are 128 plans / 14 families (A–M = 118/13 frozen v1); no dangling ROADMAP_v0.2.md links.

Manual lens

Please work §7b + the §8 sign-off checklist, and weigh in on the three §9 open questions — especially (1) is v2.0.0 the intended label (vs. keeping semver v0.2.0), and (2) is a control-tier release acceptable as v2.0.0, or should the tag wait for a primary-tier CTF run on the local Gemma-4-E2B NLA + GPU.

Reproduction (CPU, ~1 min)

pip install -r requirements.txt
python -c "import nla_eval; print(nla_eval.__version__)"   # 2.0.0
python run_example.py                                       # v1 smoke test
python experiments/ctf_red_blue_demo.py                     # v2 CTF demo (reproducible)

Note: PRs #21 (versioning scaffold) and #22 (domain-aware result) are already merged to master, so this branch already contains them — they can be closed.

---

Generated by Claude Code

[gemini-code-assist]

Code Review

This pull request bumps the benchmark version to 2.0.0 and introduces Family N — CTF Red/Blue, an adversarial capture-the-flag evaluation paradigm for NLA-as-monitor threat models. It adds the runnable harness (nla_eval/ctf.py), an offline CPU-only demo (experiments/ctf_red_blue_demo.py), and updates the plan catalog and documentation to reflect the new 128-plan structure. The review feedback highlights three key areas for improvement in nla_eval/ctf.py: correcting a logic bug in BlueMonitor.inspect where best_hit can be overwritten by a non-tripping concept, dynamically calculating the threshold in _first_clean_query using 1.0 - ceiling instead of hardcoding it to floor, and replacing float('nan') with None in score_ctf to prevent invalid JSON serialization.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[SolshineCode]

All six review threads addressed in 3e57882 and resolved:

• HIGH — invalid JSON (nan): score_ctf now returns None (→ null) for undefined catch_rate/mean_queries_to_capture; CTFScore hints updated to Optional. (Same class as the project's H3.)
• HIGH — BlueMonitor.inspect hit consistency: tripped concepts are now tracked separately, so a flagged verdict's hit is always a concept that actually hard-matched (reports the highest-soft tripping concept).
• HIGH — _first_clean_query threshold: now 1.0 - ceiling (was floor), so queries-to-first-clean is correct for any flag_ceiling.
• MEDIUM — demo formatting / newline: metrics are None-safe and 2-dp; the JSON artifact ends with a newline.

One deviation: the demo-formatting suggestion's inline {x:.2f if ... else 'N/A'} isn't valid f-string syntax (the conditional can't live in the format spec), so I used a small f2() helper that does the same thing. Validated: artifact re-parses as strict JSON, hit-consistency + threshold unit checks pass, numbers unchanged at the default floor/ceiling = 0.5.

---

Generated by Claude Code