Repository navigation
Conversation
`_extract_parenthesized_choice` searched for `\(?(\w)\)`, which also matches the final letter of any word followed by a closing parenthesis. A reply such as "I think (probably) the answer is (b)" therefore yielded "y" instead of "b". Require a word boundary before the choice character so only a standalone character such as "(b)" or "b)" is accepted.
Contributor
Author
|
Note on the red |
jzleibo
approved these changes
Oct 8, 2026
copybara-service Bot
pushed a commit
that referenced
this pull request
Oct 8, 2026
-- 1e7d9c4 by Anugrah <anugrahsengar92@gmail.com>: Stop choice extraction matching the last letter of a parenthesized word `_extract_parenthesized_choice` searched for `\(?(\w)\)`, which also matches the final letter of any word followed by a closing parenthesis. A reply such as "I think (probably) the answer is (b)" therefore yielded "y" instead of "b". Require a word boundary before the choice character so only a standalone character such as "(b)" or "b)" is accepted. COPYBARA_INTEGRATE_REVIEW=#399 from anugrah55:fix-choice-extraction-word-boundary 1e7d9c4 PiperOrigin-RevId: 996144444 Change-Id: Ie496b0c895bce19e6ea9815f5a506a8b15382ffa
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
sampling.extract_choice_responseis what most model wrappers use insample_choiceto pull the chosen letter out of a reply. For replies longer than two characters it searches for\(?(\w)\), which also matches the last letter of any word that happens to be followed by):Usually the stray letter is not one of the offered choices, so the attempt is discarded and retried at a higher temperature. When the question has enough options that the stray letter is a valid key (for example
ewith five or more choices), the wrong answer is accepted without any error.Fix
Require a word boundary before the choice character (
\(?\b(\w)\)), so only a standalone character such as(b)orb)is accepted. The three replies above now give'b','a'andNone.Callers are unchanged. The one- and two-character fast paths in
extract_choice_responseare not touched.Tests
Three cases added to
sampling_test.py. They fail on currentmainand pass with the change; the existing cases are unaffected.Run on macOS arm64, Python 3.13.9, on top of
026de0c:pytest concordia: 1061 passed, 33 skippedpylint --errors-onlyandpyrefly checkon the changed files: clean