fix(inspect): align GenerateConfig field names with current inspect-ai (fixes #1327) - #1328
fix(inspect): align GenerateConfig field names with current inspect-ai (fixes #1327)#1328sk8ordie84 wants to merge 2 commits into
Conversation
lighteval eval forwards its generation options to inspect_ai_eval_set(), which passes anything it does not declare itself on to GenerateConfig. Three of those names are not GenerateConfig fields in current inspect-ai, so the Inspect-backed eval path raises a pydantic ValidationError at run time: frequence_penalty -> frequency_penalty log_probs -> logprobs response_format -> no longer a GenerateConfig field The first two are now mapped at the call site, leaving the CLI parameters unchanged. --frequency-penalty is added alongside the existing (misspelled) --frequence-penalty so both spellings work; --log-probs is untouched, since a Python field named logprobs does not require the flag to be renamed. response_format is forwarded only when the installed GenerateConfig accepts it, with a warning otherwise, so the flag keeps working on older inspect-ai and is ignored rather than fatal on newer. Fixes huggingface#1327
Reads the keyword names out of the inspect_ai_eval_set(...) call in main_inspect.py rather than restating them, so it keeps checking the real call site as that call site changes. Any name that inspect_ai.eval_set does not declare must be a field the installed GenerateConfig accepts. Fails on main today, naming frequence_penalty, log_probs and response_format.
b038d69 to
5c19f66
Compare
|
Status update, and a fix to my own patch. My original patch renamed the CLI parameters, which would have broken existing I also re-checked the underlying problem against inspect-ai 0.3.263 on Python Added a regression test. It reads the keyword names out of the Happy to split the regression test into its own PR if that would make review |
1 participant
Fixes #1327.
A fresh
pip install lightevalresolvesinspect-aito the latest release, andlighteval evalcrashes at startup for any model/task with a pydanticValidationError, because threeGenerateConfigfield names inmain_inspect.pyno longer exist upstream.Changes:
frequence_penalty→frequency_penalty(typo; matches the inspect-ai field name)log_probs→logprobs(matches the inspect-ai field name)response_formatis passed through only when the installed inspect-ai still accepts it (GenerateConfig.model_fieldscheck); otherwise it is ignored with a warning. The field was removed upstream in inspect-ai ≥ 0.3.141, so unconditionally passing it can never work there.Verified on inspect-ai 0.3.257 (Python 3.12):
lighteval eval mockllm/model "lighteval|gsm8k|0" --max-samples 4 --log-dir out) now completes and writes results;--frequency-penalty 0.5 --logprobsare accepted and forwarded;--response-format '{"type":"object"}'no longer crashes (ignored with a warning on current inspect-ai).Note: the two renamed CLI flags were unusable on current installs (startup crash), so the rename does not break any working invocation. Happy to add a deprecation alias if preferred.