post_process: per-experiment embedding step (per-marker, SLURM, reuses aggregate_channels) - #21
Merged
Merged
Conversation
Adds run_per_exp_embeddings: for a single experiment, processes each CellDINO
marker on its own (no cross-experiment correction, no second-pass PCA). Reuses
aggregate_channels for all the rich outputs (UMAP/PHATE overlays + interactive
HTMLs, mAP consistency/distinctiveness bars, sweep plots, coord CSVs, gene/guide
h5ads), then adds a gene x gene correlation heatmap (PNG/SVG/interactive HTML)
and records every post-processing decision in decisions.yaml (distance,
pca_variance, norm_method, zscore, agg_method, ...) instead of encoding them in
directory names. Outputs land under cell_dino_features_v2/embeddings/<marker>/.
Filename bridge to aggregate_channels' <marker>_{guide,gene}.h5ad glob uses a
hardlink (no symlink, no data copy).
Each marker's aggregate_channels + correlation heatmap is independent, so submit one SLURM job per marker via submit_parallel_jobs (matches cell_dino_main). Only discovery + submission run locally; all compute runs on SLURM. slurm=False keeps the in-process path for local/debug runs.
GO enrichment (one pass per Leiden resolution) is the long pole in aggregate_channels, and the multi-exp default runs ~13 resolutions. Thread a leiden_resolutions param through aggregate_channels -> save_extra_overlays (which already accepts it), default None so the multi-exp pipeline keeps its full DEFAULT_LEIDEN_RESOLUTIONS. The per-exp step sets it to [4,10,30] (recorded in decisions.yaml), cutting the GO-enrichment cost to 3 resolutions.
OpsPaths.model_checkpoints_dir() derived the checkpoint root from _resolve_base() (OPS_OUTPUT_BASE_DIR). In research/rerun mode that redirects to the rerun output tree (e.g. /reruns/rerun/models/...), where the read-only model checkpoints don't exist — celldino extraction died with FileNotFoundError on dinov2/hubconf.py. Model checkpoints are inputs, so anchor them to the canonical data root (OPS_BASE_PATH, default /hpc/projects/icd.fast.ops), overridable via OPS_MODELS_BASE_DIR. General fix for any model-loading step run in rerun mode; surfaced by celldino on an ops0105 rerun copy.
The correlation heatmap was only emitted as PNG/SVG/HTML; add a labelled gene x gene corr_heatmap.csv alongside so users can download the raw values.
gav-sturm
changed the title
…lots/)
- corr_heatmap.{png,svg,html,csv} now live in a correlation_heatmap/ subdir
instead of the marker dir root.
- _organize_plots groups the flat aggregate_channels plots into category subdirs
(umap/, phate/, map_metrics/, sweep/, channel_qc/) so plots/ isn't a sprawling
list of PNGs; existing subdirs (leiden/, canonical_leiden/, marker_overlay/)
are left in place.
…bdir - gene_ebi_complex_overlay.* and gene_ebi_binary_overlay.* get their own ebi_complex_overlay/ and binary_overlay/ subdirs (were lumped into map_metrics/ via the 'ebi' match) — checked before map_metrics so they win. - Drop plots/canonical_leiden/guide/ (guide-level canonical clustering not needed as its own subdir; gene-level canonical clustering kept).
…iden/ - Single ebi_overlay/ subdir for both gene_ebi_complex_overlay.* and gene_ebi_binary_overlay.* (was two subdirs). - Move canonical_leiden/ (gene + guide) into leiden/canonical/ instead of a top-level subdir (and no longer drop the guide-level canonical clustering).
The upstream 'canonical' name is opaque; the subdir holds the GO-term-annotated cluster embeddings (top enriched GO term labelled at each cluster centroid via _save_canonical_panel_png). go_annotated is self-describing.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
New
run_per_exp_embeddings— a per-experiment embedding post-processing step that runs after CellDINO inference, one marker at a time (no cross-experiment correction, no second-pass PCA; those only matter when combining markers/experiments).What it does
aggregate_channelsso all the rich outputs are produced unchanged: UMAP/PHATE overlays + interactive HTMLs, mAP consistency/distinctiveness bars, sweep plots, coord CSVs, gene/guide h5ads.corr = np.corrcoef(X_ops - X_ops.mean(0))on the PCA-to-variance gene embedding) as PNG + SVG + interactive Plotly HTML + a downloadable CSV of the values.decisions.yaml(distance, pca_variance, norm_method, zscore, agg_method, leiden_resolutions, n_pcs, …) instead of encoding them in directory names.3-assembly/cell_dino_features_v2/embeddings/<marker>/.Key behaviours
submit_parallel_jobs(matchescell_dino_main); only discovery + submission run locally.slurm=Falsefor a local/debug path.[4, 10, 30]for the per-exp step (GO enrichment is the long pole and scales with resolution count). Threaded throughaggregate_channels → save_extra_overlays; defaultNonekeeps the multi-exp pipeline's fullDEFAULT_LEIDEN_RESOLUTIONS.OpsPaths): checkpoints now resolve from the canonical base (OPS_BASE_PATH), notOPS_OUTPUT_BASE_DIR, so model-loading steps work in research/rerun mode. General fix (also unblocks celldino in rerun mode).aggregate_channels'<marker>_{guide,gene}.h5adglob uses a hardlink (no symlink, no data copy).Validated end-to-end
Run on 3 experiments via the pipeline runner, all on SLURM:
decisions.yamlwithleiden_resolutions: [4,10,30]; runtime ~11–15 min/marker (vs. the 13-resolution default).Diff vs. the archived real ops0105 outputs
cell_seg,gfp_segidentical;phase2d_tubular_segforeground identical (IoU 1.0, only instance-ID relabeling).nucleoli_phase2d_segdiverges (rerun ~2.5× more objects, IoU 0.61) — likely org-seg code/param evolution since the archived run.Driven by the pipeline via royerlab/ops_process#160's new
embedding_postprocessstep (added to the DAG aftercelldino_inference).