One-pass qmap-partial specificity metric, replacing fast and qnorm spec

[davek44]

Summary

Replaces fast spec (per epoch) and qnorm spec (hound_eval_spec, post hoc) with one streaming specificity metric. It is reported per epoch in training (spec, spec/<group> in metrics.tsv) and per track in hound_eval (spec column of acc.txt).

Changes

• hound_data: writes target_hist to each examples/*.zarr: exact whole-genome histograms of the stored fp16 targets, per track, with strand pairs summed (num_targets × 31744). Replaces write_target_means.
• metrics.SpecPearsonCorrCoef:
  • Tables: builds qnorm's tie-averaged quantile map for each track from target_hist (qmap_tables).
  • Per batch: sums strand pairs, maps preds and targets through the target's table, regresses the group mean out of each track, and accumulates moments.
  • Output: per-track Pearson r of the residuals.
• hound_eval: computes spec in the normal eval pass.
• hound_train_folds --eval: scores the test fold at the end of each training job.
• Removed:
  • hound_eval_spec, spec_norm.py, seqnn.eval(combine_pairs=...);
  • hound_eval_folds options --spec, --band, --seq_chunk, --ram and --scratch_dir.

Validation

• Agreement with qnorm spec: Spearman ρ across tracks within each group, on test sets of a 4-fold hg38 + mm10 model.
  • ChIP groups 0.97–0.99, and ≥ 0.9 in 13 of 17 groups.
  • Sparse single-cell groups are lower: 0.77–0.89.
  • For reference, fast spec gave −0.33 on hg38 H3K4me3.
• Early stopping: over 160 checkpoints, epoch-to-epoch changes in valid spec track those in test spec at r 0.94 (summed over datasets).
• Tests:
  • tables match a sort-based qnorm reference;
  • the streaming metric matches a direct numpy computation;
  • depth-only gain scores ≈ 0, and a planted track-specific signal is recovered;
  • histogram counts are exact, including strand pairs;
  • NaN predictions, near-identical tracks and constant group means are handled.

Known limitation

Predictions are mapped through the target's table, not through their own distribution as qnorm does. In sparse groups, where most bins are zero, this reads up to 0.10 lower than qnorm and ranks tracks less consistently with it.

Breaking changes

• Datasets need target_hist. Without it, spec is disabled with a warning. Backfill with python -c "from baskerville import dataset; dataset.write_target_hist('data_dir')". The mean attr is no longer written or read.
• spec changes meaning: in metrics.tsv and acc.txt it now reports the new metric.
• GCP eval resume: the run-id hash no longer includes spec options, so earlier GCP eval runs won't resume.

🤖 Generated with Claude Code