Repository navigation
checks: reviewer independence by shared evidence - #442
Merged
Merged
Conversation
A second reader catches p * acc * (1 - overlap) of the lies the first checker clears. The check measures overlap from evidence ids and returns PASS when the catch's lower bound reaches 0.05, FAIL when its upper bound is under, and UNVERIFIABLE when the interval straddles. Tests reproduce the society lane's stop points and catch. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Turns convergence 1 from the saves reading ("a second checker that reads the same evidence repeats the first one's errors") into a check, using the society lane's rule.
harness/reviewer_independence.py: overlap = share of the second reader's evidence ids the first already read. Catch =p * acc * (1 - overlap); stop point1 - 0.05 / (p * acc).pandacctake a number or a{k, n}count (Wilson 95% bounds). Returns aflywheel.check-receipt/v2: PASSsecond_reader_payswhen the catch's lower bound reaches the floor, FAILsecond_reader_redundantwhen its upper bound is under, UNVERIFIABLE when the interval straddles or an input is missing. CLI:python -m harness.reviewer_independence case.json.CHECKSregistry, because registry kinds need a false-accept corpus entry set; that can follow.docs/features/reviewer-independence.md. CHANGELOG under Unreleased.Verified:
tests/test_reviewer_independence.py: 10 passed locally (Python 3.12, Windows). They reproduce the society table: stop points 0.69 / 0.75 / 0.79 at acc 0.65 / 0.8 / 0.95 for p 0.247, catch 0.100 at overlap 0.5, and the fresh-look stop point (0.393 from p 0.103; the table prints 0.40 from an unrounded p).check_file_gate.py,check_verifier_stdlib.py: clean.Not covered: correlation through a shared model family with no shared evidence (H143), and the planted spec-weakening test for our own gates (H144).
🤖 Generated with Claude Code