Skip to content

Return nan instead of raising when ignore_index removes every sample - #3464

Open
arose26 wants to merge 3 commits into
Lightning-AI:masterfrom
arose26:fix/ignore-index-all-samples
Open

Return nan instead of raising when ignore_index removes every sample#3464
arose26 wants to merge 3 commits into
Lightning-AI:masterfrom
arose26:fix/ignore-index-all-samples

Conversation

@arose26

@arose26 arose26 commented Aug 18, 2026

Copy link
Copy Markdown

Fixes #2685

Curve-based classification metrics raise IndexError when ignore_index filters out every sample:

preds = torch.tensor([0.1, 0.8, 0.4])
target = torch.tensor([-1, -1, -1])
BinaryAUROC(ignore_index=-1)(preds, target)
# IndexError: index is out of bounds for dimension with size 0

The same happens in the multilabel case when a single label column is entirely ignore_index while the other labels still have samples — which is the more common way to hit this, since per-label curves are computed independently:

preds = torch.tensor([[0.9, 0.5, 0.2], [0.1, 0.3, 0.8], [0.7, 0.6, 0.4]])
target = torch.tensor([[1, -1, 0], [0, -1, 1], [1, -1, 0]])
MultilabelAUROC(num_labels=3, average=None, ignore_index=-1)(preds, target)
# IndexError

AUROC, AveragePrecision, ROC and PrecisionRecallCurve are affected, in their binary, multiclass and multilabel forms. Only the non-binned path — with thresholds set, the state is a confusion matrix and _safe_divide already returns a value instead of raising.

Cause

After filtering, preds and target reach _binary_clf_curve empty. With no samples, target.size(0) - 1 is -1:

threshold_idxs = F.pad(distinct_value_indices, [0, 1], value=target.size(0) - 1)
tps = _cumsum(target * weight, dim=0)[threshold_idxs]

so threshold_idxs is tensor([-1]), which indexes an empty cumsum.

Change

An undefined score is reported as nan. That is already how this library treats a class it cannot score: _reduce_auroc and _reduce_average_precision warn "score for one or more classes was nan. Ignoring these classes in {average}-average" and drop those entries via idx = ~torch.isnan(res). The binned path likewise uses _safe_divide(..., zero_division="nan"). So macro and weighted averages keep working over the labels that do have data, and only average=None surfaces the nan.

Guarding the crash alone is not enough. If _binary_clf_curve merely returns empty tensors, _binary_roc_compute prepends its zero point and torch.trapz over a single point returns 0.0 — a legitimate-looking "perfectly wrong classifier" score standing in for "no data". Returning a wrong number is worse than raising, so the guard is on the curve having fewer than two points, not on it being empty:

  • _binary_clf_curve returns empty curves instead of indexing tensor([-1])
  • _binary_roc_compute and _binary_precision_recall_curve_compute return nan for an empty curve (the latter also avoids tps[-1])
  • _auc_compute_without_check returns nan for a curve with fewer than two points
  • the list branch of _reduce_average_precision scores such a class as nan rather than summing empty slices to -0.0

Tests

test_corner_case_all_samples_ignored and test_corner_case_single_label_ignored, added for both AUROC and AveragePrecision. The multilabel test asserts all three halves of the behaviour: the ignored label is nan, the two populated labels keep their real values, and average="macro" stays finite.

All four fail on master with IndexError and pass with this change. tests/unittests/classification/{test_auroc,test_average_precision,test_roc,test_precision_recall_curve}.py and tests/unittests/utilities/ pass: 1026 passed, 100 skipped, 16 xfailed. ruff check and ruff format --check are clean.

Note on scope

The issue asks for this to be consistent across metrics. This PR covers the metrics that actually raise. BinaryPrecision and friends already return 0.0 here through their own zero_division handling, which is a separate and configurable behaviour, so they are left alone rather than changed under the same PR.

Curve-based classification metrics indexed an empty tensor once ignore_index
filtered out every sample of a class, which is reached in normal use when one
label of a multilabel target carries no valid entries in a batch.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Make ignore_index work when all batch elements are to be ignored

1 participant