Skip to content

Fix PESQ aborting a whole batch on one unscorable sample - #3455

Open
Kayvan-Zahiri wants to merge 1 commit into
Lightning-AI:masterfrom
Kayvan-Zahiri:fix/pesq-degenerate-sample-3304
Open

Fix PESQ aborting a whole batch on one unscorable sample#3455
Kayvan-Zahiri wants to merge 1 commit into
Lightning-AI:masterfrom
Kayvan-Zahiri:fix/pesq-degenerate-sample-3304

Conversation

@Kayvan-Zahiri

Copy link
Copy Markdown

Fixes #3304.

One unscorable sample takes down the whole batch. pesq and pesq_batch both
default to on_error=PesqError.RAISE_EXCEPTION and torchmetrics never passes
on_error, so what happens depends on n_processes:

  • n_processes=1 (the default): pesq() raises NoUtterancesError and it
    propagates out of update(). The _filter_error_msg guard added in Ignore the NoUtterancesError when calculating pesq for a batch #2753 is
    unreachable on this path.
  • n_processes != 1: pesq_batch() catches worker exceptions and returns them
    in the result list, so the guard is reached, but it drops the failed
    entry. A 3-sample batch silently returns 2 values.

Measured on a 3-sample batch with target[1] silent:

functional,   n_processes=1  -> raises NoUtterancesError
functional,   n_processes=2  -> shape (2,)  values [2.5199, 2.1842]   <- one sample vanished
class metric, n_processes=2  -> compute()=2.3521  total=2             <- 3 in, 2 counted

After:

functional,   n_processes=1  -> shape (3,)  values [2.5199, nan, 2.1842]
functional,   n_processes=2  -> shape (3,)  values [2.5199, nan, 2.1842]
class metric, either         -> compute()=2.3521  total=2

Change

Pass on_error=PesqError.RETURN_VALUES at all three call sites and map every
failure report onto nan, keeping one entry per input sample. Failures arrive
two ways and both are handled: negative error codes (-1..-7, which cannot
collide with a real score since valid PESQ is >= -0.5) and the exception
objects pesq_batch collects from its workers.

The class metric excludes nan samples from both the running sum and total,
so a degenerate sample does not poison an epoch's average. That matches what the
multiprocessing path already did by dropping them, so it is not a new policy.

Scores for scorable samples are unchanged: RETURN_VALUES and
RAISE_EXCEPTION return the same value on success (verified,
2.158228635787964 both ways).

Also in this PR

#2753 replaced pesq_val.reshape(preds.shape[:-1]) with
reshape(len(pesq_val)), so a (2, 3, 2100) input returned (6,) instead of
(2, 3), contradicting the documented (...,) shape. Restored, which is safe
now that no entries are dropped. This is a behaviour change for ndim > 2
inputs
relative to 1.4.x-1.9.0. Happy to split it into its own PR if you would
rather keep this one minimal.

Tests

Four cases in tests/unittests/audio/test_pesq.py, all failing on main:

test_degenerate_sample_does_not_abort_batch[1]  NoUtterancesError
test_degenerate_sample_does_not_abort_batch[2]  assert torch.Size([2]) == (3,)
test_degenerate_sample_returns_nan_for_single_sample  NoUtterancesError
test_multidim_input_keeps_shape                 assert torch.Size([6]) == (2, 3)

The two different failure modes for [1] and [2] are the two paths above.

tests/unittests/audio/test_pesq.py: 20 -> 24 passed, same 6 skipped and 3
xfailed. Doctests pass. Wider audio dir: 88 passed. ruff, format and mypy clean.

The pesq backend defaults to on_error=PesqError.RAISE_EXCEPTION, so a
single degenerate sample (e.g. a silent reference, or a prediction whose
amplitude dwarfs the reference after the backend's shared normalisation)
raised NoUtterancesError out of the whole update.

Pass on_error=PesqError.RETURN_VALUES at every call site and map the
returned error codes, as well as the exception objects pesq_batch
collects from its workers, onto nan. The result now keeps one entry per
input sample, so the documented shape contract holds again for inputs
with more than one batch dimension. The class metric leaves nan samples
out of its average instead of propagating them.

Fixes Lightning-AI#3304
@Kayvan-Zahiri
Kayvan-Zahiri force-pushed the fix/pesq-degenerate-sample-3304 branch from e87534b to 8426d03 Compare August 24, 2026 18:35
@mergify mergify Bot removed the has conflicts label Aug 24, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

PerceptualEvaluationSpeechQuality - NoUtterancesError: b'No utterances detected'

1 participant