Eval V2: Code-eval scorer helpers + custom-typed metrics (re-landing of #1568) - #1592

Open

chiang-daniel wants to merge 23 commits into

mainfrom

dchiang/eval-helpers-custom-scores

Commits on Jul 17, 2026

fix(evals): get_tool_results never matched real traces

committed
a37551d
fix(evals): reject NaN/inf scores at validation and in the code-eval adapter

committed
fc36685
fix(evals): skip custom-typed scores in eval config correlation summary

chiang-daniel
and
claude
committed
89639ee
feat(evals): KilnEvalHelpers.get_tool_result_content

committed
9b6cbef
feat(evals): KilnEvalHelpers.get_markdown_links

committed
7ae2ed3
feat(evals): KilnEvalHelpers.count_messages

committed
3d3a66f
feat(evals): KilnEvalHelpers.get_assistant_emitted_text

committed
ca0764a
feat(evals): usage, latency, and tool-error helpers for scorers

committed
f3f9305
feat(evals): custom-typed eval output scores (code-eval-only metrics)

committed
a068049
fix(evals): make get_markdown_links linear on adversarial input

chiang-daniel
and
claude
committed
87fedee
fix(evals): custom scores require code-eval configs, not just non-judge

chiang-daniel
and
claude
committed
498797a
fix(evals): robustness polish from the phase's adversarial review

chiang-daniel
and
claude
committed
0c70a80
test(evals): pin get_usage_totals to MessageUsage.from_trace

chiang-daniel
and
claude
committed
4b5bdb9

Commits on Aug 19, 2026

Merge sfierro/KIL-741 into eval-helpers-custom-scores

chiang-daniel
and
claude
committed
9125390
Fix import order flagged by CI ruff after the KIL-741 merge

chiang-daniel
and
claude
committed
dd84a52
Harden eval helper text and usage aggregation

chiang-daniel
and
claude
committed
f590ce8
Keep custom metrics out of run config ranking

chiang-daniel
and
claude
committed
1696578
Block custom-typed scores at create_evaluator

chiang-daniel
and
claude
committed
5517f73
Custom-score API errors: refuse earlier, say why

chiang-daniel
and
claude
committed
6aee0c9
Scorer helper and custom-metric display fixes

chiang-daniel
and
claude
committed
562f84c
get_tool_results: drop "tool_result" from the role match

chiang-daniel
and
claude
committed
67a1c4a

Commits on Aug 20, 2026

Make these tests pin the behaviour their names claim

chiang-daniel
and
claude
committed
51b2cc8
Merge sfierro/KIL-741; resolve get_tool_results to this branch's stricter role match

chiang-daniel
and
claude
committed
62f7ed4