Skip to content

fix(llm): keep evaluator-only config out of the request - #488

Merged
seancoding-day merged 1 commit into
MigoXLab:devfrom
seancoding-day:dev
Sep 2, 2026
Merged

fix(llm): keep evaluator-only config out of the request#488
seancoding-day merged 1 commit into
MigoXLab:devfrom
seancoding-day:dev

Conversation

@seancoding-day

Copy link
Copy Markdown
Collaborator

Evaluator config and provider request parameters share one bag — EvaluatorLLMArgs is extra="allow", so both land in model_extra — and everything in it was forwarded to chat.completions.create. The SDK takes no unknown keyword arguments, so configuring a documented knob such as threshold (RAG, agent_eval, instruction_quality) or strictness made every call by that evaluator raise TypeError, which the caller then shapes into an ordinary "evaluation failed" — indistinguishable from a timeout.

What may be sent is now decided by the SDK's own create() signature rather than a list someone has to remember to update. A denylist was tried first and missed two keys on the first pass; the signature travels with the SDK. Keys that are neither request parameters nor registered local knobs are dropped with a warning, so a misspelled key no longer fails silently.

llm_custom_metric forwarded model_extra directly and so bypassed the filter entirely; it now goes through the same accessor. request_timeout and max_retries become configurable per evaluator, the latter reaching the client rather than the request body. Paths that had no timeout before only get one when configured — inventing a default there would fail long calls that work today.

Evaluator config and provider request parameters share one bag —
``EvaluatorLLMArgs`` is ``extra="allow"``, so both land in ``model_extra``
— and everything in it was forwarded to ``chat.completions.create``. The
SDK takes no unknown keyword arguments, so configuring a documented knob
such as ``threshold`` (RAG, agent_eval, instruction_quality) or
``strictness`` made every call by that evaluator raise TypeError, which
the caller then shapes into an ordinary "evaluation failed" —
indistinguishable from a timeout.

What may be sent is now decided by the SDK's own ``create()`` signature
rather than a list someone has to remember to update. A denylist was
tried first and missed two keys on the first pass; the signature travels
with the SDK. Keys that are neither request parameters nor registered
local knobs are dropped with a warning, so a misspelled key no longer
fails silently.

``llm_custom_metric`` forwarded ``model_extra`` directly and so bypassed
the filter entirely; it now goes through the same accessor. ``request_timeout``
and ``max_retries`` become configurable per evaluator, the latter reaching
the client rather than the request body. Paths that had no timeout before
only get one when configured — inventing a default there would fail long
calls that work today.
@seancoding-day
seancoding-day merged commit 5bae389 into MigoXLab:dev Sep 2, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant