Skip to content

Prevent integer overflow when computing spectral angles - #186

Open
antonsoo wants to merge 2 commits into
spectralpython:developfrom
antonsoo:fix/spectral-angle-norm-accumulation
Open

antonsoo wants to merge 2 commits into
spectralpython:developfrom
antonsoo:fix/spectral-angle-norm-accumulation

Conversation

@antonsoo

Copy link
Copy Markdown

Accumulate image-vector squared norms in at least float64. Currently the einsum uses the image dtype, so ordinary int16 values from read_bands can overflow before the square root. This produces NaNs or incorrect angles even though the same spectra work when explicitly converted to float64, as discussed in #132.

Use np.result_type to retain wider floating precision. The existing float64 endmember normalization and dot-product calculation stay the same.

Add 14 scaled-direction cases covering signed/unsigned integers and floating types. All 16 spectral-angle tests pass on Python 3.8/NumPy 1.24 and Python 3.12/NumPy 2.5. The five numerical test files pass 165 tests, and the required Flake8 check passes.

On the official AVIRIS sample, all 12 dtype/read-path configurations now match an independent geometric angle reference and its nearest-endmember decisions. The three float64 controls remain byte-for-byte identical.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant