fix: lazy polars import for Ivy Bridge CPU compatibility#117
Open
LuJiansen wants to merge 1 commit into
Open
Conversation
Make polars imports lazy in counting/ modules to prevent 'Illegal instruction' crashes on Intel Ivy Bridge (Xeon E5-2670 v2) nodes that lack AVX2/AVX-512 support. Changes: - counting/count_alleles.py: * Add TYPE_CHECKING guard for pl.DataFrame annotation * Lazy import polars inside make_count_df() * Add make_count_df_no_polars() using pure Python csv + Rust counting - counting/run_counting.py: * Route non-gene paths through make_count_df_no_polars * Remove unused parse_intersect_region_new import * Remove unused region_col_name assignments - counting/filter_variant_data.py: * Add TYPE_CHECKING guard + lazy import in gtf_to_bed() - counting/parse_gene_data.py: * Add TYPE_CHECKING guard + lazy import in parse_gene_file() - counting/count_alleles_sc.py: * Add TYPE_CHECKING guard + lazy import in make_count_matrix() ruff check + ruff format: clean
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
On Intel Ivy Bridge CPUs (Xeon E5-2670 v2), the
polars-runtime-32shared library crashes withIllegal instructionbecause it is compiled with AVX2/AVX-512 instructions that these CPUs lack. This affects bothwasp2-map make-readsandwasp2-count count-variants.Clusters with mixed CPU architectures (Ivy Bridge + Haswell in the same Slurm partition) cannot use
--constraintto avoid these nodes.Solution
Make
polarsimports lazy — moveimport polars as plfrom module top-level into the functions that actually use polars. AddTYPE_CHECKINGguards so static analysis can still resolvepl.DataFrameannotations without triggering a runtime import.For
count-variants(the most common counting path), add a polars-free alternativemake_count_df_no_polars()using pure Pythoncsv+ the existing RustBamCounter.Files changed (5 files, +101 −29)
src/counting/count_alleles.pyTYPE_CHECKINGguard + lazy import + newmake_count_df_no_polars()src/counting/run_counting.pymake_count_df_no_polars; remove unused imports and dead assignmentssrc/counting/filter_variant_data.pyTYPE_CHECKINGguard + lazy importsrc/counting/parse_gene_data.pyTYPE_CHECKINGguard + lazy importsrc/counting/count_alleles_sc.pyTYPE_CHECKINGguard + lazy importWhat is NOT broken
make_count_df()(polars path) still works on CPUs with AVX2 supportmapping/intersect_variant_data.pywas already lazy-imported — no changes neededTesting
ruff check: All checks passedruff format --check: 5 files already formattedpy_compile: 5/5 files pass_parse_intersect_tsvunit tested