Skip to content

[Feature] Support DROP exact-match and F1 metrics - #2616

Open
Excelius-Wang wants to merge 1 commit into
open-compass:mainfrom
Excelius-Wang:feat/drop-f1-score
Open

[Feature] Support DROP exact-match and F1 metrics#2616
Excelius-Wang wants to merge 1 commit into
open-compass:mainfrom
Excelius-Wang:feat/drop-f1-score

Conversation

@Excelius-Wang

Copy link
Copy Markdown

Thanks for your contribution and we appreciate it a lot. The following instructions would make your pull request more healthy and more easily get feedback. If you do not understand some items, don't worry, just make the pull request and seek help from maintainers.

Motivation

The OpenAI-derived DROP evaluator currently reports only fuzzy substring accuracy. For example, it treats Denver as fully correct for Denver Broncos, although the standard DROP exact-match score is 0 and F1 is 66.67.

Adding the official metrics makes OpenCompass results comparable with other published DROP evaluations while retaining the existing accuracy output.

Closes #948.

Modification

  • add the standard DROP single-span normalization, exact-match, and F1 logic, including numeric normalization and number matching
  • take the best EM and F1 over the pipe-delimited alternative reference answers
  • preserve the existing accuracy result and add exact_match and f1 aggregates and per-example details
  • add focused tests for text and numeric normalization, empty inputs, mismatched numbers, partial overlap, alternative answers, and legacy result compatibility

The simple-evals DROP data stores each pipe-delimited reference as an alternative single-span answer, so this implements the equivalent single-span path of the official metric without adding a SciPy dependency for multi-span alignment.

BC-breaking (Optional)

No. The existing accuracy key and correct detail field are unchanged; the new metrics are additive.

Use cases (Optional)

Users can report standard DROP EM/F1 directly from the existing drop_openai_simple_evals configuration and compare OpenCompass runs with published model results.

Checklist

Before PR:

  • Pre-commit or other linting tools are used to fix the potential lint issues.
  • Bug fixes are fully covered by unit tests, the case that causes the bug should be added in the unit tests.
  • The modification is covered by complete unit tests. If not, please add more unit test to ensure the correctness.
  • The documentation has been modified accordingly, like docstring or example tutorials.

After PR:

  • If the modification has potential influence on downstream or other related projects, this PR should be tested with those projects.
  • CLA has been signed and all committers have signed the CLA in this PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature] Support F1-score for DROP dataset

2 participants