Skip to content

Fix chrF dropping the reference n-grams of a zero-scoring sentence - #3481

Open
Kayvan-Zahiri wants to merge 1 commit into
Lightning-AI:masterfrom
Kayvan-Zahiri:fix-chrf-zero-score-reference-ngrams
Open

Fix chrF dropping the reference n-grams of a zero-scoring sentence#3481
Kayvan-Zahiri wants to merge 1 commit into
Lightning-AI:masterfrom
Kayvan-Zahiri:fix-chrf-zero-score-reference-ngrams

Conversation

@Kayvan-Zahiri

Copy link
Copy Markdown

What does this PR do?

Fixes #3480

_calculate_sentence_level_chrf_score starts best_f_score at tensor(0.0) and only records a reference under if f_score > best_f_score. A hypothesis that shares no character or word n-gram with any of its references scores exactly 0.0, so the guard never fires and the function returns its zero-filled defaults. _chrf_score_update then adds those zeros to the corpus reference totals while still adding the hypothesis n-grams, so the sentence drops out of the recall denominator and the corpus score is inflated. On the example in the issue that is 0.7080 against sacrebleu's 0.4217, and it makes the metric non-monotonic: garbage can outscore a partly correct translation.

The fix keeps the first reference unconditionally, which is what sacrebleu does with its best_f_score = -1.0 sentinel. A better-scoring later reference still wins, so multi-reference behavior is unchanged.

Two tests, both verified to fail on master: one checks the corpus score against _reference_sacrebleu_chrf, the other checks that degrading a hypothesis cannot raise the score.

Before submitting
  • Was this discussed/agreed via a Github issue? (no need for typos and docs improvements)
  • Did you read the contributor guideline, Pull Request section?
  • Did you make sure to update the docs? (no docs change needed, the documented behavior was already the intended one)
  • Did you write any new necessary tests?
PR review

Anyone in the community is free to review the PR once the tests have passed.
If we didn't discuss your PR in Github issues there's a high chance it will not be merged.

Did you have fun?

Chasing a metric that rewards worse translations was a good one, yes.

`_calculate_sentence_level_chrf_score` starts `best_f_score` at 0.0 and only
records a reference under `if f_score > best_f_score`. A hypothesis that shares
no character or word n-gram with any of its references scores exactly 0.0, so
the guard never fires and the function returns the zero-filled defaults.
`_chrf_score_update` then adds those zeros to the corpus reference totals while
still adding the hypothesis n-grams, so the sentence drops out of the corpus
recall denominator and the corpus score is inflated.

Keep the first reference unconditionally, which is what sacrebleu does with its
`best_f_score = -1.0` sentinel.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

chrF drops the reference n-grams of a zero-scoring sentence, inflating the corpus score

1 participant