Skip to content

Native methylation-aware phasing: wire the ported algorithm into make_examples#6

Open
nh13 wants to merge 4 commits into
IPNP-BIPN:apple-silicon-native-v2-prfrom
nh13:nh_dv-5-methylation
Open

Native methylation-aware phasing: wire the ported algorithm into make_examples#6
nh13 wants to merge 4 commits into
IPNP-BIPN:apple-silicon-native-v2-prfrom
nh13:nh_dv-5-methylation

Conversation

@nh13

@nh13 nh13 commented Jun 23, 2026

Copy link
Copy Markdown

Stacked on #7 (which is stacked on #5) — review after they merge. Opt-in (--enable_methylation_aware_phasing, default off); default behavior is byte-identical. Apple-only native code; nothing ports upstream.

Depends on #7 for its prerequisites (aux-field parsing + the single-sample multi-sample VariantCaller). This PR is now just the methylation-aware phasing feature itself.

What this does

Wires the already-ported, unit-tested methylation_aware_phasing algorithm into the make_examples single-sample phasing block (mirroring make_examples_core.py): methylated reference sites (alt='.') are split out of the SNP phasing graph, DirectPhasing runs on the SNP candidates, and PerformMethylationAwarePhasing then assigns a haplotype (Wilcoxon rank-sum vote on 5mC) to reads DirectPhasing left unphased. The refined per-read phases feed read_hp_tags / the haplotype small_model exactly as the DirectPhasing output does.

  • The option is set on AlleleCounterOptions/VariantCallerOptions/MakeExamplesOptions.
  • dv_methylation_aware_phasing is linked into dv_make_examples_lib.
  • A log line reports methylated-ref-site count + reads rephased when the path engages.

Scope: single-sample path (the primary PacBio/ONT methylation use case); per-site p-values are dropped (upstream uses them only for --phasing_error_stats_output). Like --enable_methylation_calling, the flag is exposed at the make_examples stage, not forwarded through run.

Validation

  • 14/14 ctest green (incl. the upstream methylation_aware_phasing unit test run against the native build). Flag off (default): byte-identical.
  • Real PacBio HiFi 5mC data (PacBio's public HG002-CpG-methylation-202202 BAM, chr20 ~40×, streamed via a .bai-guided byte-range fetch): with Native single-sample correctness: multi-sample VariantCaller + aux-field parsing fix #7's prerequisites in place, the active path engages — 6 methylated reference sites per region, PerformMethylationAwarePhasing runs on real 5mC. (At 40× HiFi DirectPhasing already phases all reads, so the vote correctly finds 0 additional reads to rephase; the algorithm executing on real methylation data is the end-to-end proof. A sparser-SNP locus would exercise the rephase-count path.)
    • Note: the public dataset uses legacy lowercase Mm/Ml tags (pre-2021 SAM convention); DeepVariant expects uppercase MM/ML. Renaming them (semantically identical) is required for that specific file — modern PacBio/ONT outputs use the uppercase tags.

@BenjaminDEMAILLE

Copy link
Copy Markdown

Heads-up @nh13: I had to squash apple-silicon-native-v2-pr from 273 commits down to 1, because Google's CLA bot can't auto-scan PRs with more than 250 commits and this branch is the head of the upstream PR google#1085. The squash is content-identical (same tree hash 810ff855), only the history was flattened.

Side effect: this rewrote the base of your 4 open PRs (#3, #4, #6, #7), so they now show an inflated diff against r1.10. A rebase onto the new base fixes it cleanly (no real conflicts, the content is unchanged). The old base tip was 0cd41f859de445f5d8a60524b86b54bb418a043b, so:

git fetch origin
git rebase --onto origin/apple-silicon-native-v2-pr 0cd41f859de445f5d8a60524b86b54bb418a043b <your-branch>
git push --force-with-lease

Sorry for the churn, and thanks again for all the fixes. Your authorship is preserved via a Co-Authored-By trailer on the squashed commit.

nh13 added 4 commits June 24, 2026 11:15
--parse_sam_aux_fields was set on MakeExamplesOptions but never on the
SamReaderOptions used to open the BAM, and sam_reader.cc's ParseAuxFields is a
no-op unless aux_field_handling == PARSE_ALL_AUX_FIELDS. So the flag did
nothing: no aux fields (MM/ML base modifications, HP, ...) were ever parsed
into the Read proto. Set aux_field_handling = PARSE_ALL_AUX_FIELDS on sam_opts
when the flag is on; default off keeps the byte-identical baseline (WGS chr20
unchanged at 182 examples; 14/14 ctest green).

Found while validating methylation-aware phasing on real PacBio HiFi 5mC data:
without this, read.base_modifications() is always empty so the 5mC path can
never engage. (It also gates BAM HP-tag parsing used by the phasing fallback.)
The single-sample path used vcf_candidate_importer::VariantCaller
(variant_calling.cc), which does not emit methylated reference sites
(alt='.'); only multi_sample::VariantCaller (variant_calling_multisample.cc)
does. Upstream always routes through the multi-sample VerySensitiveCaller for
every sample count — vcf_candidate_importer is reserved for the special
--variant_caller=vcf_candidate_importer import mode. So methylation-aware
phasing could never get methylated_ref_sites in the single-sample path.

Switch the single-sample path to multi_sample::VariantCaller (matching the
trio path and upstream), calling CallsFromAlleleCounts with a single-element
{sample_name -> AlleleCounter*} map and role 'sample'. This is the change that
makes methylation-aware phasing actually engage.

Validated on the chr20 HG002 WGS fixture (full chr20, b37, vs GIAB v4.2.1):
F1 0.9974 -> 0.9975, recall 0.9955 -> 0.9957, precision 0.9993 unchanged
(57 FP) — the broader candidate set recovers a few true positives the
narrower importer caller missed, with no precision cost, i.e. it is also more
upstream-faithful. On real PacBio HiFi 5mC data the methylated-ref-site path
now engages (6 methylated ref sites/region; PerformMethylationAwarePhasing
runs). 14/14 ctest green. Apple-only; nothing ports upstream.
The methylation_aware_phasing algorithm was compiled + unit-tested but never
invoked by the native pipeline; --enable_methylation_aware_phasing did not
exist, so the feature was unreachable. Port upstream's integration
(make_examples_core.py): when the flag is set, AlleleCounter extracts 5mC
levels and the VariantCaller emits methylated reference sites (alt='.') as
candidates; in the single-sample phasing block those sites are split out of
the SNP phasing graph, DirectPhasing runs on the SNP candidates, and
PerformMethylationAwarePhasing then assigns a haplotype (Wilcoxon rank-sum
vote on 5mC) to reads DirectPhasing left unphased. The refined per-read phases
feed read_hp_tags / the haplotype small_model exactly as the DirectPhasing
output does. The option is set on AlleleCounterOptions, VariantCallerOptions,
and MakeExamplesOptions; dv_methylation_aware_phasing is now linked into
dv_make_examples_lib.

Scope: single-sample path only (the primary PacBio/ONT methylation use case),
matching how the other phasing features are wired; the per-site p-values are
dropped (upstream uses them only for --phasing_error_stats_output, not ported).
Like --enable_methylation_calling, the flag is exposed at the make_examples
stage and not forwarded through the `run` orchestrator.

Validation: build + 14/14 ctest green (incl. the methylation_aware_phasing
unit test). Flag off (default) is byte-identical — the option is false on all
three option protos and the phasing block reduces to phasing the full
candidate set as before. Flag on, chr20 HG002 (Illumina, no 5mC): runs clean,
182 examples (same as off) — the methylation path is correctly inert without
modification data. NOTE: the active 5mC phasing path is NOT validated on
aligned methylation data (no PacBio/ONT methylation BAM fixture is available);
its correctness rests on the algorithm's unit tests + structural parity with
upstream. Opt-in and gated. All changes are Apple-only native code.
When --enable_methylation_aware_phasing engages the 5mC vote, log the number
of methylated reference sites and reads rephased per region, so the active
path is observable in run logs. Fires only when the flag is on and methylated
ref sites are present (no per-region noise otherwise).
@nh13 nh13 force-pushed the nh_dv-5-methylation branch from c4b614a to ec06b4b Compare June 24, 2026 18:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants