Skip to content

Reporting faithfulness: audit items B1-B11 (plan B) - #75

Open
druvus wants to merge 14 commits into
fix-audit-data-safetyfrom
fix-audit-report-faithfulness
Open

druvus wants to merge 14 commits into
fix-audit-data-safetyfrom
fix-audit-report-faithfulness

Conversation

@druvus

@druvus druvus commented Oct 1, 2026

Copy link
Copy Markdown
Contributor

Implements plan B of docs/superpowers/specs/2026-10-01-post-1.2.0-audit-design.md (items B1-B11): outputs that misstated what a run did.

Stacked on #74 (plan A, fix-audit-data-safety). This PR targets that branch; retarget it to main once #74 is merged, and merge #74 without deleting its branch first.

What changes

  • Run provenance names each caller correctly (MaxChi, Bootscan, GENECONV and barcode were all recorded as "heuristic") and records the agreement gate, sibling exclusion, lineage clustering and donor re-attribution.
  • A coverage gap produced by windows straddling a called breakpoint is labelled breakpoint; it no longer caveats the region or lowers the confidence wording. example_data/divergent.msa.fasta now reports high confidence, as its README says.
  • The report's "covering N kb" is the union of overlapping regions.
  • recombination_methods.tsv and the method table follow --reattribute-donors.
  • --lineage-map to a missing file is an error in five commands.
  • A barcode caller that cannot run is not run (with the reason), does not count toward --min-methods, and a barcode-only run on such a panel is refused.
  • PHI: the report uses the run's --alpha; an alignment where the window covers every site pair is not testable (phi_p is None, NA in the profile header) instead of "no signal".
  • Plots: donor-absent bands, the pair plot (major vs leading donor), colours with --top-n 1.
  • Stale report, help and documentation text; seaborn dropped; the sibeliaz version probe no longer records an error line; remaining CLI input checks.

What does not change, and the exceptions

For a given alignment the callers find the same regions with the same coordinates and p-values. Exceptions: donor_undercovered and donor-absent rows where a gap is now a breakpoint gap; runs that select barcode on a panel where it cannot run.

Checks (this branch, on top of plan A)

  • ruff, mypy: clean. pytest -m "not requires_binary": 717 passed (also with a CI-like PATH and COLUMNS=80). requires_binary: 5 passed.
  • validation/run_specificity.py --reps 3: 0/12 runs with a false region at --min-methods 2; 7/12 (8 regions, all HMM) at --min-methods 1; positive control 3/3 detected, 3/3 donor, 55 bp at both. Identical to main.
  • validation/run_validation.py (mafft, minimap2): 6 PASS, 1 FAIL (hcv_clonal_1b, a 12 bp MaxChi-only region, as on main), 1 SKIP (orthopox_example). validation/README.md is corrected to say so.

Not verified -- please read before merging

  • validation/run_hybrids.py was not run (it needs network). The breakpoint relabelling can change its panel_donor_absent scoring: a region whose donor is a close stand-in for an absent lineage (above the coverage threshold) used to be caveated only through its breakpoint gaps and is now uncaveated. On synthetic data this happened in 4 of 12 such cases. Run the harness on main and on this branch before merging.
  • Breakpoint relabelling under informative-site windowing: no adversarial case was built.

Review

One independent whole-branch review; fixes in 0c9daf2. Its main finding: the first version of the breakpoint rule compared the two parents with the query over the whole gap, which let a short stretch from a source outside the panel (150 bp at 15 % divergence beside a breakpoint) be relabelled. The rule now requires every under-threshold window in the gap to be matched, to the threshold, by a single switch between the region's two parents.

Deferred minor findings:

  • the provenance "coverage threshold / gaps" count includes breakpoint gaps while the headline excludes them;
  • in run_benchmark.py / run_coalescent_benchmark.py, window-invariant alignments move from "tested, p = 1" to "untestable", so earlier power/specificity figures are not directly comparable; their docstrings still say "too few informative columns";
  • docs/detection-methods.md describes the barcode not run case only for an untyped panel;
  • after --reattribute-donors the region names the clade's first member, which the breakpoint check then uses; a real breakpoint artefact may stay divergent (the conservative direction).

🤖 Generated with Claude Code

druvus and others added 14 commits October 1, 2026 23:01
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…n probe

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…he real not-run reason

The whole-branch review found that the breakpoint relabelling could hide a
missing reference, and three statements that were stronger than the code.

- A gap was relabelled when the two parents together matched the query over
  the whole gap. Averaged that way a short stretch from a source outside the
  panel is diluted by its well-matching flanks: 150 bp at 15 % divergence
  beside a breakpoint was relabelled and the report read "high confidence".
  Each under-threshold window must now be matched, to the threshold, by a
  single switch between the two parents -- the coverage question itself, with
  a mosaic in place of one reference.
- On a typed panel the provenance and report said barcode "needs typed
  references"; they now carry the reason the log gives.
- A --min-methods gate lowered because a selected caller could not run is
  logged and recorded with the requested value.
- The PHI "not testable" wording stated a condition that was off by one.
- The changelog no longer says "no region call changes" without its exceptions.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant