Skip to content

Stop choice extraction matching the last letter of a parenthesized word - #399

Open
anugrah55 wants to merge 1 commit into
google-deepmind:mainfrom
anugrah55:fix-choice-extraction-word-boundary
Open

anugrah55 wants to merge 1 commit into
google-deepmind:mainfrom
anugrah55:fix-choice-extraction-word-boundary

Conversation

@anugrah55

Copy link
Copy Markdown
Contributor

sampling.extract_choice_response is what most model wrappers use in sample_choice to pull the chosen letter out of a reply. For replies longer than two characters it searches for \(?(\w)\), which also matches the last letter of any word that happens to be followed by ):

>>> from concordia.utils import sampling
>>> sampling.extract_choice_response('I think (probably) the answer is (b)')
'y'
>>> sampling.extract_choice_response('maybe) (a)')
'e'
>>> sampling.extract_choice_response('(yes) a')
's'

Usually the stray letter is not one of the offered choices, so the attempt is discarded and retried at a higher temperature. When the question has enough options that the stray letter is a valid key (for example e with five or more choices), the wrong answer is accepted without any error.

Fix

Require a word boundary before the choice character (\(?\b(\w)\)), so only a standalone character such as (b) or b) is accepted. The three replies above now give 'b', 'a' and None.

Callers are unchanged. The one- and two-character fast paths in extract_choice_response are not touched.

Tests

Three cases added to sampling_test.py. They fail on current main and pass with the change; the existing cases are unaffected.

Run on macOS arm64, Python 3.13.9, on top of 026de0c:

  • pytest concordia: 1061 passed, 33 skipped
  • pylint --errors-only and pyrefly check on the changed files: clean

`_extract_parenthesized_choice` searched for `\(?(\w)\)`, which also matches
the final letter of any word followed by a closing parenthesis. A reply such
as "I think (probably) the answer is (b)" therefore yielded "y" instead of
"b".

Require a word boundary before the choice character so only a standalone
character such as "(b)" or "b)" is accepted.
@anugrah55

Copy link
Copy Markdown
Contributor Author

Note on the red Test examples check: it fails in the "Typecheck examples" step on examples/concordia_island/run.py:647 (Tensor returned where np.ndarray is declared), a file this PR does not touch. The same step has failed on every push to main since 2026-10-06, so it is unrelated to this change. Test Concordia and the other checks pass.

copybara-service Bot pushed a commit that referenced this pull request Oct 8, 2026
--
1e7d9c4 by Anugrah <anugrahsengar92@gmail.com>:

Stop choice extraction matching the last letter of a parenthesized word

`_extract_parenthesized_choice` searched for `\(?(\w)\)`, which also matches
the final letter of any word followed by a closing parenthesis. A reply such
as "I think (probably) the answer is (b)" therefore yielded "y" instead of
"b".

Require a word boundary before the choice character so only a standalone
character such as "(b)" or "b)" is accepted.

COPYBARA_INTEGRATE_REVIEW=#399 from anugrah55:fix-choice-extraction-word-boundary 1e7d9c4
PiperOrigin-RevId: 996144444
Change-Id: Ie496b0c895bce19e6ea9815f5a506a8b15382ffa

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants