feat(fix-gaps): blind Jev ids + Noul scoring, v4 validation (gate fail, stays experimental)

CHANGELOG.md

@@ -7,6 +7,19 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+### Changed
+- **Breaking (experimental section-mode JSON only):** section mode now scores each candidate with an independent Jev Noul question instead of one Choice question (owner decision D2, openspec `update-jev-blind-validation`). In `fix-gaps.json` the page `diagnosis` reports `scorer: "noul"` and `scores` in place of `probabilities` / `confidence`; `verification` reports `score_before` / `score_after` in place of `p_before` / `p_after`. `already_best` and `won` require the best own section to score strictly above every source (ties count as losing). Page mode, the default, is unchanged.
+- Jev now sees neutral candidate ids (`c01`, `c02`, …) in an order derived from the query and the texts, so it cannot tell your sections from the cited sources. Diagnosis and verification use the same ids.
+- `jev_validation` reports the pre-registered v4 validation of the configuration section mode runs (`protocol: "v4"`, `scorer: "noul"`, `ids: "blind"`, `unit: "sections"`): mean AUC 0.559 (95% CI 0.489–0.637, 30 queries, 2026-10-15), below the 0.65 gate, so `status` stays `"experimental"` and every Jev diagnosis and verification keeps `"experimental": true`. The 2.2.0 figures move to `jev_validation.previous`, labeled `choice, leaky ids, page excerpts`.
+- `eval/jev_selection` scores with the product function on sections (`own_source`), adds `--ids {blind,leaky}`, `--prereg-sha` (required for live runs) and `--repo-root`, and drops `--scorer`.
+- The offline mock answers Noul questions by query-word overlap instead of a constant 0.9.
+
+### Added
+- `eval/jev_selection/PREREG-v4.md` (registered 2026-10-08, commit `56ef2db`) and `queries-v3.json`: the v4 validation protocol, the query list and its hash, the gate and the consequences of passing and failing. The leak diagnostic and the v4 result are in `eval/jev_selection/README.md`.
+
+### Known limitations
+- Jev diagnosis and verification in section mode **failed** their pre-registered validation (mean AUC 0.56 against a 0.65 gate). When Jev judged the own section best, Perplexity cited the page in 1 of 6 queries. Treat `already_best` as a weak signal and confirm with `remeasure`. Section mode stays experimental and opt-in; page mode, the default, does not use Jev.
+
 ## [2.2.0] - 2026-09-26
 
 ### Known limitations

egeo/gaps.py

@@ -261,35 +261,60 @@ def run_fix_gaps(
     "rewriter_error",          # the rewriter LLM failed for this page (LLMError); nothing written
 )
 
-# The figures below are the pre-registered v2 run of the *choice* scorer: the same single Choice question
-# (judge.select_best) that section mode uses to diagnose and verify. The gate-deciding noul scorer
-# (AUC 0.637) is not what section mode runs, so its numbers are not quoted here.
+# v4 (openspec update-jev-blind-validation): section mode scores every candidate with an independent Noul
+# question under blind ids (c01, c02, ...). Its pre-registered validation (2026-10-15) failed the gate. The
+# 2.2.0 number measured a different configuration (single Choice question, ids that revealed ownership,
+# full-page excerpts), so it is quoted only as an earlier configuration, never as this version's result.
 SECTIONS_NOTE = (
     "Section mode: the Jev comparison between your sections and the sources the tracker says were cited is "
-    "EXPERIMENTAL. It scores text competitiveness (which candidate's text best answers the query) and does not "
-    "model authority or links, so it is not a prediction of citation. In a pre-registered test (30 queries, "
-    "2026-09-25) the same single-choice comparison separated cited from uncited sources with mean AUC 0.62 "
-    "(95% CI 0.56-0.67), below the 0.65 bar, and matched the engine's cited/not-cited outcome for the own page "
-    "in only 17% of the queries that had one. That test compared full-page excerpts, not sections "
-    "(eval/jev_selection/README.md). Confirm any change by re-checking the queries in `remeasure` with your tracker."
+    "EXPERIMENTAL. It scores text competitiveness (how well each candidate's text answers the query) and does "
+    "not model authority or links, so it is not a prediction of citation. The configuration this version runs "
+    "(an independent score per candidate, neutral ids that hide which text is yours, sections) scored mean AUC "
+    "0.56 (95% CI 0.49-0.64, 30 queries, 2026-10-15) in a pre-registered test, below the 0.65 bar "
+    "(eval/jev_selection/README.md). Treat `already_best` and the re-check as weak signals. An earlier "
+    "configuration (single-choice comparison, ids that revealed ownership, full-page excerpts) scored mean AUC "
+    "0.62 (95% CI 0.56-0.67, 30 queries, 2026-09-25). Confirm any change by re-checking the queries in "
+    "`remeasure` with your tracker."
 )
 
-# Copied verbatim into every section-mode report as "jev_validation"; every non-null page "diagnosis" and
-# "verification" also carries "experimental": True (owner decision 2026-09-25, plan v3).
+# Copied verbatim into every section-mode report as "jev_validation". While "status" != "validated", every
+# non-null page "diagnosis" and "verification" carries "experimental": True (one switch, flipped only by the
+# pre-registered outcome, plan v4 §4). The v4 run failed the gate, so the status stays "experimental".
 JEV_VALIDATION = {
     "status": "experimental",
-    "scorer": "choice",
-    "mean_auc": 0.619,
-    "ci95": [0.561, 0.674],
-    "own_accuracy": 0.167,
+    "protocol": "v4",
+    "scorer": "noul",
+    "ids": "blind",
+    "unit": "sections",
+    "mean_auc": 0.559,
+    "ci95": [0.489, 0.637],
+    "own_accuracy": 0.5625,
     "n_queries": 30,
-    "unit": "full-page excerpts (<= 1500 chars), not sections",
     "gate_auc": 0.65,
+    "gate_min_queries": 25,
     "gate_passed": False,
-    "date": "2026-09-25",
+    "prereg_sha": "56ef2db55125c691bd72c7f350ce8dc3dae31d27",
+    "date": "2026-10-15",
     "details": "eval/jev_selection/README.md",
+    "previous": {
+        "config": "choice, leaky ids, page excerpts",
+        "mean_auc": 0.619,
+        "ci95": [0.561, 0.674],
+        "own_accuracy": 0.167,
+        "n_queries": 30,
+        "gate_passed": False,
+        "date": "2026-09-25",
+    },
 }
 
+
+def _flag(d):
+    """Add "experimental": True to a diagnosis/verification dict while Jev is not validated."""
+    if JEV_VALIDATION["status"] != "validated":
+        d["experimental"] = True
+    return d
+
+
 # Every page dict gets an "offpage" key (plan v2). For status "already_best" it is
 #   {"reason": OFFPAGE_REASON, "message": OFFPAGE_MESSAGE, "sources": [url of every ok fetched doc, in fetch order]}
 # and for every other status it is None. The CLI line for an already_best page is
@@ -327,22 +352,19 @@ def _section_page(page, *, out_dir, jev_client, rewrite_fn, fetch_fn, exclude_do
            "offpage": None, "reasons": []}
     try:
         diag = judge.diagnose(page.query, own, srcs, jev_client)
-        if diag.selection is None or diag.status == "already_best":
-            sel = diag.selection
-            res["diagnosis"] = None if sel is None else {"status": diag.status, "winner": sel.winner, "winner_kind": sel.winner_kind,
-                                                        "confidence": sel.confidence, "probabilities": dict(sel.probabilities),
-                                                        "target_section": None, "experimental": True}
+        if diag.scored is not None:
+            lead = judge.leader(diag.scored, own, srcs)
+            res["diagnosis"] = _flag({"status": diag.status, "winner": lead.id, "winner_kind": lead.kind, "scorer": "noul",
+                                      "scores": dict(diag.scored.scores), "target_section": None})
+        if diag.scored is None or diag.status == "already_best":
             res["status"] = diag.status
             if diag.status == "already_best":
                 res["offpage"] = {"reason": OFFPAGE_REASON, "message": OFFPAGE_MESSAGE,
                                   "sources": [d.url for d in docs if d.ok]}
             return res
         sid = diag.target_id[len("own_"):]
         sec = next(s for s in secs if s.id == sid)
-        sel = diag.selection
-        res["diagnosis"] = {"status": diag.status, "winner": sel.winner, "winner_kind": sel.winner_kind, "confidence": sel.confidence,
-                            "probabilities": dict(sel.probabilities), "target_section": {"id": sid, "heading": sec.heading},
-                            "experimental": True}
+        res["diagnosis"]["target_section"] = {"id": sid, "heading": sec.heading}
         try:
             new_text = rewrite_fn(page.query, sec.text, title)
         except LLMError as exc:
@@ -358,9 +380,10 @@ def _section_page(page, *, out_dir, jev_client, rewrite_fn, fetch_fn, exclude_do
         if not v.accepted:
             res["status"] = "rejected_fidelity_judge"; return res
         own_after = [judge.Candidate(c.id, c.kind, c.label, new_text if c.id == diag.target_id else c.text) for c in own]
-        ver = judge.verify(page.query, own_after, srcs, diag.target_id, sel, jev_client)
-        res["verification"] = {"p_before": ver.p_before, "p_after": ver.p_after, "winner_after": ver.winner_after,
-                               "winner_after_kind": ver.winner_after_kind, "outcome": ver.outcome, "experimental": True}
+        ver = judge.verify(page.query, own_after, srcs, diag.target_id, diag.scored, jev_client)
+        res["verification"] = _flag({"score_before": ver.score_before, "score_after": ver.score_after,
+                                     "winner_after": ver.winner_after, "winner_after_kind": ver.winner_after_kind,
+                                     "outcome": ver.outcome})
         if ver.outcome == "worse":
             res["status"] = "rejected_worse"; return res
     except (JevProviderError, JevConfigError) as exc:

egeo/jev.py

@@ -37,7 +37,9 @@
   to 4 decimals. ``choice`` = highest probability (ties -> first option in criteria order).
   ``confidence = max(0, min(1, (n * peak - 1) / (n - 1)))`` with n options (n == 1 -> 1.0), where
   ``peak`` is the UNROUNDED highest probability; round the confidence to 4 decimals.
-- noul: 0.9.
+- noul: the share of query words found in the candidate the question points at:
+  ``round(|words(query) ∩ words(candidates[qid])| / max(1, |words(query)|), 4)`` when ``state`` is a
+  dict with ``"query"`` and a dict ``"candidates"`` containing the question id ``qid``; else 0.0.
 - model "mock", usage zeros. Returns status 200.
 
 ``make_client(mock=..., model=...)``: mock -> ``JevClient(MockJevTransport(), model="mock")``;
@@ -151,7 +153,12 @@ def post_json(self, url, body):
                 answers[qid] = {"type": "choice", "choice": best, "confidence": round(conf, 4),
                                 "probabilities": {o: round(p, 4) for o, p in zip(opts, raw)}}
             else:
-                answers[qid] = {"type": "noul", "noul": 0.9}
+                cands = state.get("candidates") if isinstance(state, dict) else None
+                score = 0.0
+                if isinstance(cands, dict) and qid in cands:
+                    qw = _words(state.get("query", ""))
+                    score = round(len(qw & _words(cands[qid])) / max(1, len(qw)), 4)
+                answers[qid] = {"type": "noul", "noul": score}
         return 200, {"model": "mock", "answers": answers, "usage": {"input_tokens": 0, "output_tokens": 0}}