Skip to content

docs(auroc): clarify per-batch logit/probability check in BinaryAUROC - #3474

Open
mohansree14 wants to merge 1 commit into
Lightning-AI:masterfrom
mohansree14:docs/2195-clarify-per-batch-logit-conversion
Open

docs(auroc): clarify per-batch logit/probability check in BinaryAUROC#3474
mohansree14 wants to merge 1 commit into
Lightning-AI:masterfrom
mohansree14:docs/2195-clarify-per-batch-logit-conversion

Conversation

@mohansree14

Copy link
Copy Markdown

What does this PR do?

Fixes #2195

BinaryAUROC.update() (inherited from BinaryPrecisionRecallCurve) checks per batch whether preds looks like logits (values outside [0, 1]) and applies sigmoid if so. This check runs independently on every update() call rather than once across the full accumulated dataset, so with small batch sizes it's possible for some batches to be treated as probabilities and others as logits within the same run, producing inconsistent results.

Per the reporter's suggested resolution, this PR documents that behavior more prominently in BinaryAUROC's class docstring rather than changing the underlying per-batch logic (which other metrics also rely on), to keep this a minimal, low-risk documentation fix. Happy to extend this to the base BinaryPrecisionRecallCurve docstring too, or take a different approach, if maintainers prefer.

Before submitting

  • Was this discussed/agreed via a GitHub issue? (Conversion from logits to probabilities happens on a batch by batch basis #2195)
  • Did you read the contributor guideline?
  • Did you make sure your PR does only one thing, instead of bundling different changes together?
  • Did you make sure to update the documentation with your changes? (n/a — this PR is the doc update)
  • Did you write any new necessary tests? (n/a — documentation-only change, no behavior change)
  • Did you verify new and existing tests pass locally?
  • Did you list all the breaking changes introduced by this pull request?

This is my first contribution to this repo — feedback welcome.

Fixes Lightning-AI#2195 - the sigmoid-vs-probability check in update() runs per
batch, not across the full dataset, which can silently produce
inconsistent conversion behavior across batches. Documented this in
the class docstring.
@codecov-commenter

codecov-commenter commented Aug 21, 2026

Copy link
Copy Markdown

⚠️ Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 31%. Comparing base (d184220) to head (91b9915).
⚠️ Report is 12 commits behind head on master.
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

❗ There is a different number of reports uploaded between BASE (d184220) and HEAD (91b9915). Click for more details.

HEAD has 784 uploads less than BASE
Flag BASE (d184220) HEAD (91b9915)
torch2.0.1+cpu 20 2
python3.10 129 13
Windows 30 3
cpu 218 22
torch2.9.1 9 1
macOS 39 4
python3.12 89 9
torch2.9.1+cpu 30 3
torch2.8.0+cpu 30 3
torch2.8.0 10 1
Linux 149 15
torch2.10.0 10 1
torch2.7.1+cpu 20 2
torch2.10.0+cpu 20 2
torch2.4.1+cpu 10 1
torch2.2.2+cpu 10 1
torch2.1.2+cpu 10 1
torch2.5.1+cpu 10 1
torch2.0.1 10 1
torch2.3.1+cpu 9 1
torch2.6.0+cpu 10 1
Additional details and impacted files
@@           Coverage Diff            @@
##           master   #3474     +/-   ##
========================================
- Coverage      37%     31%     -6%     
========================================
  Files         349     349             
  Lines       19901   19910      +9     
========================================
- Hits         7264    6170   -1094     
- Misses      12637   13740   +1103     
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Conversion from logits to probabilities happens on a batch by batch basis

2 participants