[mverab]

v4 result (§3–§4) — GATE FAIL

• Dataset v3 collected once (2026-10-15, 30 queries, 0 empty answers); own_source from main @ ffd5b49.
• Evaluated once: --ids blind --prereg-sha 56ef2db. Mean AUC 0.559 [0.489, 0.637] < 0.65. Own wins × cited: tp 1, fp 5, fn 2, tn 8. Cost $0.0083.
• Per the pre-registered consequence: JEV_VALIDATION carries the v4 numbers, status: "experimental", labels stay. Note, CLI docs and CHANGELOG updated.
• Owner decision (2026-10-16): keep section mode experimental. Demoting already_best from a verdict to a warning goes to a separate OpenSpec change.

Summary

Implements openspec update-jev-blind-validation waves 0–1 (owner decision D2 = Noul, 2026-10-07).

• Blind ids: Jev sees c01, c02, … ordered by sha256(query + "\n" + text). Original ids, hosts and headings never reach it. Diagnosis and verification share one id map, computed once from the original texts.
• Noul scoring: one request per decision, with an independent Noul question per candidate. already_best / won require the best own section to score strictly above every source.
• Report (experimental JSON, breaking): diagnosis.scores + scorer: "noul" replace probabilities / confidence; verification.score_before/after replace p_before/after. JEV_VALIDATION describes v4 with no result; the 2.2.0 figures sit under previous. The experimental flags follow JEV_VALIDATION["status"].
• Eval = product path: eval/jev_selection calls egeo.judge.score_candidates and reads sections from own_source. Adds --ids {blind,leaky}, --prereg-sha (required for live runs; a run without it never gates) and --repo-root. Own metrics (confusion table, Wilson CI) count sections items only and never gate.
• Mock: Noul = share of query words in the candidate (was a constant 0.9).
• Pre-registration draft: eval/jev_selection/PREREG-v4.md + queries-v3.json (same 30 queries as v2; 16 mapped to own_source). Not registered yet.

Deviations from the plan, for review

• Spec scenario "no question text contains own or src" is reworded to "no original id, host or heading". NOUL_INSTRUCTIONS is kept verbatim per the plan, and it contains the generic phrase "own text".
• gating = ids == "blind" and bool(prereg_sha) (the plan has ids == "blind"), so a mock or unregistered run can never pass.
• Scored gains model, and each per-query result records jev_model, so the returned model string is logged per request (D4).

Not done (needs the owner)

• Approve and register PREREG-v4.md (flip its Status line; that commit's SHA is --prereg-sha)
• The collector (~/.hermes/scripts/egeo_jev_selection_collect.py) must copy own_source into dataset items
• §3 live steps: leak diagnostic on v2, collect v3 once, evaluate once
• §4 outcome: JEV_VALIDATION, site/…/docs/cli.md caution box, CHANGELOG. Do not merge before §4.

Test plan

• python -m pytest tests/ -q: 176 passed
• openspec validate update-jev-blind-validation --strict
• Offline mock eval on dataset v2 + real site sections (unit_counts: sections 16, page_excerpt 2)

🤖 Generated with Claude Code

https://claude.ai/code/session_016oShpRQ2da2eSmYifKqk8H

[mverab]

Pre-registration REGISTERED (owner approval 2026-10-08). --prereg-sha 56ef2db55125c691bd72c7f350ce8dc3dae31d27. queries-v3.json sha256 094e056c7ee12b4f75af36b57a8cfb5a7639f303bf526d8d03c0e2f5d4fb1718. No live v4 run before this commit.