Return nan instead of raising when ignore_index removes every sample - #3464
Open
arose26 wants to merge 3 commits into
Open
Return nan instead of raising when ignore_index removes every sample#3464arose26 wants to merge 3 commits into
arose26 wants to merge 3 commits into
Conversation
Curve-based classification metrics indexed an empty tensor once ignore_index filtered out every sample of a class, which is reached in normal use when one label of a multilabel target carries no valid entries in a batch.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #2685
Curve-based classification metrics raise
IndexErrorwhenignore_indexfilters out every sample:The same happens in the multilabel case when a single label column is entirely
ignore_indexwhile the other labels still have samples — which is the more common way to hit this, since per-label curves are computed independently:AUROC,AveragePrecision,ROCandPrecisionRecallCurveare affected, in their binary, multiclass and multilabel forms. Only the non-binned path — withthresholdsset, the state is a confusion matrix and_safe_dividealready returns a value instead of raising.Cause
After filtering,
predsandtargetreach_binary_clf_curveempty. With no samples,target.size(0) - 1is-1:so
threshold_idxsistensor([-1]), which indexes an empty cumsum.Change
An undefined score is reported as
nan. That is already how this library treats a class it cannot score:_reduce_aurocand_reduce_average_precisionwarn"score for one or more classes was nan. Ignoring these classes in {average}-average"and drop those entries viaidx = ~torch.isnan(res). The binned path likewise uses_safe_divide(..., zero_division="nan"). So macro and weighted averages keep working over the labels that do have data, and onlyaverage=Nonesurfaces thenan.Guarding the crash alone is not enough. If
_binary_clf_curvemerely returns empty tensors,_binary_roc_computeprepends its zero point andtorch.trapzover a single point returns 0.0 — a legitimate-looking "perfectly wrong classifier" score standing in for "no data". Returning a wrong number is worse than raising, so the guard is on the curve having fewer than two points, not on it being empty:_binary_clf_curvereturns empty curves instead of indexingtensor([-1])_binary_roc_computeand_binary_precision_recall_curve_computereturnnanfor an empty curve (the latter also avoidstps[-1])_auc_compute_without_checkreturnsnanfor a curve with fewer than two points_reduce_average_precisionscores such a class asnanrather than summing empty slices to-0.0Tests
test_corner_case_all_samples_ignoredandtest_corner_case_single_label_ignored, added for both AUROC and AveragePrecision. The multilabel test asserts all three halves of the behaviour: the ignored label isnan, the two populated labels keep their real values, andaverage="macro"stays finite.All four fail on
masterwithIndexErrorand pass with this change.tests/unittests/classification/{test_auroc,test_average_precision,test_roc,test_precision_recall_curve}.pyandtests/unittests/utilities/pass: 1026 passed, 100 skipped, 16 xfailed.ruff checkandruff format --checkare clean.Note on scope
The issue asks for this to be consistent across metrics. This PR covers the metrics that actually raise.
BinaryPrecisionand friends already return0.0here through their ownzero_divisionhandling, which is a separate and configurable behaviour, so they are left alone rather than changed under the same PR.