Eval Builder V2: spec-driven evals with generated data, real traces, and calibrated judges - #1731
Open
chiang-daniel wants to merge 417 commits into
Open
chiang-daniel wants to merge 417 commits into
chiang-daniel wants to merge 417 commits into
Conversation
The save fallback (no eval_id/spec_id to redirect to) previously showed a hand-rolled emoji check with 'Spec saved'. It now uses the centered completion card the git-import flow ships: large success check, 'Eval Created' heading, subtitle, wide primary back to evals. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
New local streaming route that re-scores previously driven conversations with a given judge prompt: reloads each chain's stored trace from disk by leaf run id and fans the existing judge unit over them, emitting the same frame contract as review_pipeline (no drive or turn frames). The judge unit (judge call, retries, per-case failure isolation, batch-fatal abort) is extracted to a shared JudgeStreamBase so the two streams cannot drift; review_pipeline behavior is unchanged. Missing or trace-less chains fail per case while the batch continues. Route tests + regenerated web schema. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Multi-turn save with judge disagreements no longer refines silently. It now runs an explicit calibration round: refine the judge from the grades, re-judge every driven conversation via the judge_traces stream, then re-review a subset picked by information value (previously disagreed first, then flipped results, then fresh stratified checks), grades reset so nothing unseen ships. Capped at two rounds; the gate target adapts to re-judge shortfall and an empty round surfaces as a retryable failure. At the cap or on a refine failure, a banner offers saving with the judge whose results were actually reviewed, noting persistent verdict churn as a sign the eval description may be ambiguous. The review CTA reads Refine Judge (with an explanatory tooltip) whenever disagreements exist. Single-turn behavior is unchanged. Round and flip telemetry, tests. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
At the round cap the review's primary button now reads Save with current judge and opens a confirmation dialog, replacing the top-of-review banner and the dead Refine Judge button that state left behind. The refine failure variant gains a real Try again action that re-runs the refine with the same grades and never consumes an improvement round, with simplified copy (failure detail stays in telemetry only). The verdict-churn hint is removed from user-facing copy pending review of the concept; flip counts remain in telemetry and subset selection. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…/multiturn-megabranch: train/val/test splits across TaskRun and EvalInput sources. PR #1662 is still open; these commits dedupe when it lands on main and syncs through.
…): the Eval V2 splits data model arrives — 9.4.2 item 0c. #1662 still open; dedupes when it lands on main and syncs through. Runner collapsed onto the split contract with tombstone re-keying on ItemKey; API layer on the shipped accessors with multi-turn count re-expressed per split; UI reads via the TS split accessor; wizard save rides the datamodel shim until native splits adoption.
…ne tombstone safety collect_tasks_for_eval_config_eval built its already-run and superseded maps from every record on the eval config, including task_run_eval ones — so a golden TaskRun scored as a test item was silently never calibrated, and a test-lane tombstone could ride (and be deleted by) a calibration job. Both maps now consider calibration records only. Also: restore the positive-case multi_turn_item_count tests dropped in the splits merge, add an end-to-end summary assertion for EvalInput-backed evals, have require_golden_set_or_422 return the narrowed filter id (un-inlining its one caller), and correct a stale comment that claimed a deleted defensive skip still existed.
…brating a non-dataset item The judge-comparison summary iterated every record on an eval config, including task_run_eval ones — a golden item scored as a test item entered the judge-vs-human stats with a fresh generation's score correlated against the stored item's human rating, and inflated percent-complete. Mirror of the collection-side rule: only eval_config_eval records count. Also: EvalRun now rejects eval_config_eval records carrying an eval_input_id (calibration compares against human ratings, which only dataset items carry), and UpdateEvalRequest.train_set_filter_id is typed DatasetFilterId so an invalid filter id is a 422 at request validation instead of a 500 in the handler.
The harness unpacked the spec tag helper into three fields (it now returns four) and read the deleted eval_input_filter_id field. It now checks the eval's test split directly: EvalInput-backed, with the expected tag filter — a stronger assertion than the field equality it replaces. Both paid tests green against live models.
The multi-turn save fork constructed its Eval through the datamodel's migrate_eval_input_filter_id shim (an undeclared kwarg plus a ty-ignore). It now writes the splits model directly: the EvalInput-backed test split in the constructor, and the TaskRun-backed train split homed via set_split so old clients and the project zip keep reading it from the legacy flat field. Saved bytes are unchanged on both arms; new tests pin the on-disk shape and its reload round-trip, where a dict-assignment mistake would be invisible in memory. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Every write-site is on the splits model, and all 67 internal eval.kiln files carrying the legacy key were migrated in place (a surgical JSON rewrite; the key becomes the equivalent splits.test entry). The fold validator and the tests whose subject was the shim go with it — the both-fields rejection lived inside the fold, and the only legacy shape any build ever wrote now fails loudly through validate_splits. Test fixtures author EvalInput-backed test splits directly in splits. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Steve's rulings on the loop: no round cap (any N is arbitrary — aligned or not), and the escape drops its confirm dialogs for the data-guide idiom, a grey Save Without Refining Further link under the primary CTA, offered only where that CTA renders (last trace, gate met) so it never sits under a Next button. Refine failures render inline with refine-vocabulary messages, the CTA retrying and the link standing; the failure note survives grade edits and clears on history navigation. The calibration_escape state, cap constant, and both dialogs are deleted; an opted-out telemetry event (rounds, disagreements, whether a refine error was showing) replaces the cap-hit signal. Grade-flip churn gets no special UI per the same thread. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The V1 spec builder refused to create an eval until the judge agreed with the human on every example. The v2 wizard's single-turn arm regressed that to one silent refine at save — refined judge shipped unreviewed, every failure swallowed. It now runs the same loop as multi-turn: refine on disagreement, re-judge every example through review_traces with the refined judge, re-grade, repeat until aligned or the reviewer saves without refining further. Single-turn reviews everything, so rounds have no subset math; a round commits only with a verdict for every example (first pass now holds the same coverage guard), per-case failures land on a retryable error that also names the save-without exit, a null judge fails loud on the refine CTA, and the loop telemetry carries the arm. No server changes: review_traces already accepts any judge. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The single-turn wizard save froze its eval slice as a hundred eval-tagged task runs carrying a server-generated output. The runner never judged those outputs — it re-runs each input fresh per run config — so the stored output was a misleading artifact that also made the slice look like reviewable data. The slice is now minted as single-turn EvalInput items and the saved eval carries an EvalInput-backed test split natively, matching the multi-turn arm; train stays in its legacy flat field and golden is untouched. The transient review judge's throwaway eval is authored the same way, through splits. Saved-bytes tests pin both arms: on-disk splits, the reload round trip, the slice contents, and rollback when a save fails after the slice is written. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The single-turn arm's eval data was generated toolless on kiln_server; this lands the machine that replaces it. A new single_turn_pipeline SSE route runs the task ONCE per generated input on the target run config — tools live, the user's keys — and judges each result locally, the one-turn sibling of review_pipeline: same frame contract (a new "run" stage joins "drive"/"judge"), same retry/timeout/per-case-failure posture as the drive runner, same batch lifecycle (runs persist batch-tagged via save-time default_tags; replace_batch_tags deletes superseded batches only after replacements exist, fail-closed on any run another flow claimed). The judge scores the input/output pair — exactly what the saved final_answer eval judges — while the run's structured trace rides the frames for the UI only. The three streams now share one drain loop: JudgeStreamBase owns events() with producer and judge-view hooks, so review_pipeline, judge_traces, and the new route cannot drift. The drive request's target-config and replace-tags fields are extracted into shared bases for the same reason. Judge authoring drops its multi-turn pin: author_judge derives trace_type from the task's turn mode, one authoring path for both arms. The batch planner gains the single-turn guidance text (wired next wave). Route tests cover the stream contract, retry/timeout/abort isolation, tagging, cost banking, and request validation; helper tests pin the fail-closed batch delete. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The wizard's single-turn arm swaps off the kiln_server one-shot generation (clarify_spec's ~100 toolless pairs, 10 shown, the rest discarded) and onto the plan-first local flow the multi-turn arm already uses. Step 4 now plans via batch_plan with single-turn guidance, mounts the shared /generate batch-plan approval table (third consumer, drive-honest labels: Planned Test Inputs, Run Task on N Inputs), mints one real input per approved prompt locally through the /generate input-batch endpoints on a new input-generator model lane, and streams the single_turn_pipeline route — the task runs once per input with tools live on the user's keys, each result judged locally. Progress, stop banners, per-case failure surfaces, and the preflight all reuse the multi-turn machinery, with the target-config resolution extracted into one shared helper and a third preflighted lane. Claims are now lazy on both arms behind the shared claims gate. Batch lifecycle reaches the single-turn arm: runs are batch-tagged, re-runs pass superseded tags for delete-on-redrive, and the draft persists the plan, the minted inputs (cache keyed on plan + generator model; never a partial mint), the lanes, and the tag bookkeeping — a reload restores to the plan screen without re-spending. Transitional, on purpose: single-turn save refuses with an explanation — its writer still expects the retired generation session and moves onto the batch-tagged local runs next, with the stratified subset review. The calibration loop's re-review keeps working unchanged over the pipeline-produced results. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
With the single-turn sibling landed, the multi-turn stream's name is the odd one out — and "review" was always the wrong emphasis for a route that drives and judges (review happens after, on its own surfaces). The pair is now symmetric: multi_turn_pipeline and single_turn_pipeline. Route path, request/run class names, client URL, tests, and schema follow; judge_traces and review_traces keep their names — they say what they do. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Wave D-iii of the single-turn bring-up — the wizard is now at full pipeline parity with multi-turn: - Pooled stratified review both arms: the single-turn drive picks the judge-stratified N//4 subset (select_review_subset) instead of review-all, and the save gate is the shared reviewed-count >= target. - One calibration loop: single-turn re-checks move onto its durable batch-tagged runs over judge_traces, which now serves both arms (a single-turn task reloads each run and judges the stored I/O pair — the same final_answer reading as its pipeline and its saved eval). The all-or-nothing re-review machinery and the review_traces route die with their last consumer. - Save writer rework: spec_with_copilot gains a single_turn arm (batch_tag + reviewed run refs + inputs). Reviewed runs become golden with ratings, feedback, and claim reviews; unreviewed REAL runs fill train; the eval slice is inputs-only EvalInputs from the driven inputs. Nothing is generated at save time on the wizard arms — the legacy sdg_session_config arm remains for the v1 manual flow. The shared run writers are renamed arm-neutral and the wizard's save-refusal stub dies. - Dataset grounding rider: the single-turn arm auto-picks a task sample (5-star preferred; machine-generated runs excluded so the planner is never grounded on its own output) and folds it into the batch planner's and input generator's data-guide params, persisting it on the Spec for provenance and in the wizard draft. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Live-verification finding: the wizard's suggested eval name is deterministic over the spec text, so a second eval on a task with a similar description regenerated an already-taken name — and the collision only surfaced as the save guard's 409, after the paid generation and the human review. New local-server route available_spec_name resolves a candidate against the task's existing specs using the same derived-tag comparison the save guard enforces (case/spacing-insensitive), returning the nearest suffixed variant on a collision (trimmed to the short-name limit, never fabricating forbidden characters). The wizard resolves suggested names through it before prefilling at Steps 1 and 3, and blocks the Step 3 advance with an inline error naming the available variant when a typed name is taken — a collision now costs nothing. The check is best-effort: if it cannot run, the save-time 409 remains the backstop. Also regenerates the agent-policy annotations, which had drifted from the routes: the pipeline renames and the review_traces removal never landed there, and the new pipeline routes were missing entirely — CI's annotation check would have failed on all of them. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…troys Live-verification follow-up to the early name check. Two coupled fixes: The wizard's review streams no longer carry the eval name. The transient review judge scores under a constant draft key (Meets Spec / meets_spec) instead of a name-derived one — nothing pre-save depends on the name anymore, so it stays freely editable until save, where it binds once as the spec identity, score column, and tag namespace. The three stream requests (both pipelines and the re-judge) drop spec_name and its validator; their extra=forbid configs make stale clients fail loudly. The batch tag remains the transient identity of the session's data on disk, as before. Accepted trade, documented on the transient config: the shipped judge's score key carries the real name while calibration used the draft key — same rubric, same verdict vocabulary. The staleness check becomes a non-destructive gate. Results judged under edited spec text previously got cleared the moment the check fired — unrecoverable even by reverting the edit. Now a derived flag blocks the review render with revert-or-discard actions and clears nothing: reverting the description restores the review exactly, and only the explicit discard (or a re-drive / new plan) throws results away. Deriving the flag also closes the browser-Forward path into a stale review, and the flag reads the spec text reactively — a function call would be invisible to Svelte's compile-time dependency tracking and freeze the gate across edits. The name is out of the snapshot entirely: renames never invalidate a review. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Five orphaned lines sat inside onMount: a dangling replaceState and goto_step with two extra closing braces, left behind when a dev-only block was stripped at commit time. The file was a syntax error, so a fresh checkout could not lint, check, or build the web UI. Local working trees carried the complete dev-only block, which is why every local build kept passing. Deleting the residue restores the exact prior onMount flow; no behavior change. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Each multi-turn synthetic eval input now carries its own drive config (synthetic-user model, provider, turns), stamped at save from the choice that actually drove the batch. With the persona and first message this makes every item a self-contained recipe: it re-drives identically under any eval that references it, which is what makes cross-eval trace reuse keyed on item ids sound. The eval-level field is gone, not demoted. The runner reads items only: an eval whose multi-turn items all lack a config fails loudly up front; a stray unstamped item skips with a clear per-item reason while the rest run. The save previously recorded whatever the model picker showed at save time, which could drift after the batch ran; it now records the choice captured when the batch was driven. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…refresh) Brings the #1662 final head (single-home splits, allocation spread, tolerant eval list, Test Dataset renames), the #1673 run/score-split specs, and the #1676 implementation (eval traces are TaskRuns, EvalRun becomes a score pointer record, TraceIndex reuse, migrate-eval-runs CLI). The earlier #1662-head merge (e7d366a) deduped cleanly. Conflict resolution centers on delete protection meeting the multi-turn cascade delete: a single delete 409s on an eval-needed target before cascading, an eval-needed ancestor stops the cascade like a live sibling branch, bulk delete reports protected runs per-run while still cascading the rest, and the cascade's run scan includes eval-generated runs so child counts can see them. Delete-protection tests now resolve through task_and_run_from_id; the multi-turn skip test builds its chained run in its own directory so it cannot leak into the runner task's runs scan. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Traces become plain TaskRuns and EvalRun becomes a score pointer record (scored_run_id + eval_usage), with TraceIndex reuse, the migrate-eval-runs CLI, delete protection, the joined run-results view, and single-home splits (flat filter fields are input-only and saved null). Our line's machinery rides on top: multi-turn re-drive and chain-leaf scoring replace upstream's blanket multi-turn skip (still transient drives and inline records this landing; the pointer-model conversion follows), tombstone/superseded recovery and the calibration dedupe filter thread through the new collectors, and cascade delete meets trace protection (409 before cascade, eval-needed ancestors stop the sweep, scans include eval-generated runs). Follow-through in the same landing: copilot eval creation writes both splits into `splits` directly, saved-bytes pins assert the single-home shape, obsolete inline single-turn runner tests are superseded by the pointer-form suites, the eval-list envelope (evals + load_error_count) is unwrapped at its consumers, and api_schema regenerates from the merged server. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The eval-trace delete-protection tests resolve through task_and_run_from_id, the loader the merged delete endpoint actually uses, so the guard is exercised instead of a 404. The multi-turn skip test builds its chained run in its own directory: a sibling task file in the same tmp_path shares the runs/ folder, and the chained run would leak into the runner task's scans and fail the single-turn task's turn-mode validation. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… whole Content-part traces made every validated TaskRun deepcopy-unsafe: pydantic's lazy content iterator cannot be pickled, so a cache-warm bulk load (the run-results trace join) crashed the whole request. TaskRun now materializes trace content at validation, which also fixes silent data loss: content parts on user and tool messages used to persist as empty lists. The join's loaders take a readonly path since they only read. TraceIndex treats a corrupt trace file like a missing one: warn, drop the entry, regenerate - one bad file no longer fails its key forever. The eval list logs each unreadable eval file with its path, so a corrupt file is distinguishable from a version mismatch. Task.runs applies the eval-generated filter before the leaf filter, so an eval-generated child chained onto a dataset run cannot hide the curated parent. Run-result rows with a missing trace say so instead of rendering blanks, beside the input the server resolves from the dataset item. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A driven conversation now persists as one standalone TaskRun: the full trace, the seed as input, eval_source naming the dataset item, and the run config on its output source so the reuse index finds it again. Its usage is honestly assistant-only at conversation totals; the synthetic user driver's spend rides the new TaskRun.synthetic_user_usage field beside it, never blended. Judging flows through the shared persist seam, so score records are pointers, scoring-time skips still name the trace, and judge cost lands in eval_usage. Scoring a stored dataset conversation points at its leaf without stamping or copying anything. The inline-record writers, the JSON trace serializer, and the blended cost fold are deleted. Records written by builds that skipped multi-turn outright become recoverable for seeded items instead of freezing them out. The cost rollup sums usage and synthetic_user_usage null-tolerantly and reads chain leaves at conversation totals from cumulative_usage; migrated legacy traces, whose blend is fused into usage, read unchanged. The paid harness now asserts persistence and reuse instead of transience. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A driven conversation that stopped short of its item's turn count, or did not end with the assistant answering, is treated as a failed generation: the drive raises before anything is saved, the job retries, and exhausted retries surface as a visible error. Never knowingly save a broken conversation. The trace index vets what it serves: the seed takes the first healthy candidate rather than the first found, and every serve re-checks the file against the requesting item's drive config, so an incomplete conversation (a migrated partial, or a file replaced by sync) is passed over and regenerated instead of being scored by every judge forever. Health is structural completeness relative to the item, never a stored flag; conversations containing failed tool calls are healthy, since judging error handling is a first-class use case. Rejected files are left on disk untouched, with the reason in the log line. Run configs that answer in two messages per turn (chain of thought prompts on models without their own reasoning step) are refused up front at readiness, before any spend: they cannot hold a multi-turn conversation, and previously degraded silently by leaking the reasoning instruction into the simulated user's script. The strategy decision is shared with the adapter so the two cannot drift. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The single-turn arm drops v1's RefineSpec surface, with its example fields and two-column diff, and shares the multi-turn form: eval name plus issue description with per-field refinement notes. The refine request now declares only the rendered field, so the copilot is never invited to draft example content; edits for undeclared fields are discarded before they can reach the saved spec and reported to telemetry. Name validation gates Next so an invalid name cannot ride through generation only to fail at save, and restored drafts filter to rendered fields so pre-change drafts cannot smuggle unseen values in. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Eleven deletion-only commits: unused exports, types and an icon the branch added and later stopped using, never-set props on the models dropdown and the Kiln Pro plan tables, an unused test parameter, and one shared task_response tool name pinned to libs/core. No behaviour change. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Output's `after` slot, added last commit, rendered inside the `.relative` root the Show All gradient anchors to. The gradient is `absolute bottom-0`, so growing that root with a footer moved the gradient below the fold it is meant to cover: it painted over the link, and carried the Show All button down into the link's row with it. The slot is gone and the caller renders the link itself, a sibling after the panel. Nothing is lost by deleting rather than reworking Output's box model — show_border defaults false and the overview never set it, so the footer already drew below everything Output paints. The two sit in a gap-1 wrapper, which holds the link 4px under the panel; the overview column's own gap-3 would have pushed it to 12. Claim review - The claim cell centres on its row, like the two cells beside it. It is usually the tallest, so this only shows when it is not: one collapsed line beside two buttons, where a top-aligned claim floated above the decision it belongs to. - The graded pill is held to 60%, leaving the case open now as the only full-strength mark on the line. Success screens - Completed drops align_title_left, so its title centres under the checkmark it has always drawn above it. Intro centres by default and 20 of its 23 callers take that default; Completed was the only left-aligner that renders an icon, and the other two carry titles long enough to wrap, which centring does read badly on. Every title Completed is given is a short phrase, and six screens get this: Fine Tune Created, Eval Created, Data Added, Prompt Optimization Job Started, and both Data Guide Saved. The full-trace test asserted the old placement and failed on the move, so it now pins the new one: outside Output's root, the panel's next sibling, in a gap-1 wrapper. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
An assistant message with no content and no tool calls is a transient provider glitch that a repeated call almost always resolves, but the retry classifier treated it as a permanent failure, so one empty reply from any lane lost the whole case (measured at three cases in eighty on the default builder path) while the UI reported every failure as retried. It is now retried like a timeout, and a content-filter refusal stays permanent because repeating it returns the same answer. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
chiang-daniel
marked this pull request as ready for review
September 15, 2026 23:59
…s with it The only drift was the suggested_name docstring on the refine-spec-with-name output, carried over from the server's own docstring change. openapi-python-client 0.29.0, matching the committed client byte for byte everywhere else. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
chiang-daniel
changed the base branch from
feat/multiturn-megabranch
to
main
September 16, 2026 00:30
This was referenced Sep 16, 2026
Open
Open
…from both sides) Merge-notes: ml_model_list.py: kept main's suggested_for_evals/suggested_for_data_gen flags and the branch's suggested_for_synthetic_user flag on the same eleven KilnModelProvider entries. Also added suggested_for_synthetic_user to the Fireworks provider of GLM 5.3 Flash, which main introduced after the branch flagged that model on its other providers; the branch's test requires every live provider of a suggested model to carry the flag. No other file touched by the resolution.
CI fails the merge PR on any bare print call in a Python file. The sandbox scorer test floods stdout on purpose, so it now writes through sys.stdout; the paid pipeline test reports its golden-set agreement through a logger. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2 tasks
test: keep two test files clear of the debug detector
2 tasks
The batch plan surface on /generate labelled its regenerate button "New Batch Plan". Sharing that component with the eval builder renamed the button to "Refine Plan" for both flows, which changed the shipped /generate page. The label is now a prop that defaults to the /generate wording, and the eval builder passes "Refine Plan" so the wizard keeps reading the way it was designed to. The eval creation end-to-end tests still assumed the old entry page. That page now opens on the Kiln Pro offer for anyone without a Copilot connection, and the template picker sits behind "Set Up Manually", so each of the five tests takes the manual branch before it picks a template. One of them also asserted the judge templates heading, which is now "LLM Judge Templates" so it reads apart from the new assistant section above it. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
test: fix the six failing Playwright tests on the eval builder branch
When the page's call to the local studio server failed or ran past its ten-second limit, the evals list page showed one error line although the evals had already loaded. The page no longer asks whether Copilot is connected, because nothing on it needs the answer: the Create button follows the stored draft alone, and the builder handles a disconnected user with its own connect card. The click analytics read the shared connection store at click time. Two changelog comments in the same function and test are cut to the fact they carried. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…t-check eb-v2 bug: eval list is hidden when copilot check fails
Every failed or aborted drive in the eval builder strands a batch, and the next drive is asked to delete all of them. The request model refused more than twenty, so after twenty-one retries every drive was rejected and the draft could not be driven again. The cap had no stated reason; a user's own stranded batches are theirs to clean, however many. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
eb-v2 bug: eval builder gets stuck after too many failed drives
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Eval Builder V2 and the multi-turn tasks it runs on, into main in one PR (Steve's call: one merge, one round of testing). 408 commits, 433 files, conflict-free against main. Pairs with Kiln-AI/kiln_server#258, which has to reach prod before this ships.
What does this PR do?
Writing a good eval by hand takes an afternoon, so most people don't, and the ones that exist are graded by a judge nobody checked. The builder turns a one-paragraph description into a spec, generates the data, runs it for real, and has a human calibrate the LLM judge on real traces before anything ships.
What it adds:
Since the first draft of this description (126 commits): the bug bash fixes from Steve, Mike, Leonard, Sam and my own walks; Steve's UI passes on every step, including the claim review restyle; case generation chunked to the server's per-call cap; empty model replies retried instead of failing the case; the debug capture removed; a dead-code sweep; and the val split from KIL-798 dealt at save.
Multi-turn tasks
Leonard's multi-turn work rides in here rather than through a separate hub PR: tasks can run as conversations, with a chat-style run page, forking from an assistant message, cumulative usage across turns, and the run chain model underneath. It was reviewed and approved by Steve in #1409 and has been the base of every eval builder walk since June. 59 of the commits are his.
Review
The branch is split into seven review-only views on the current tip, one per area, each with inline explanations on every hunk that matters: #1778 datamodel, #1782 eval runner and core, #1784 synthetic user, #1785 desktop and server routes, #1786 shared UI, #1787 existing pages, #1788 builder pages. Nothing merges from them; fixes land here.
Mechanical changes, and why
app/desktop/studio_server/api_client/plusapi_schema.d.ts: the kiln_server client regenerated against the server branch that ships with this (Kiln-AI/kiln_server#258). Not hand-edited; excluded from the review views.tight→inline={true}edits in 25 files. The warning control'stightprop meant two things at once, drop the outer margin and narrow the icon gap. The margin moved to the container, so the flag now means one thing and was renamed to say so. Same meaning at every call site.trace.svelte(189 real lines of 409),streaming_chat.ts(31 of 185),available_models_dropdown.svelte(56 of 160). The real lines are explained inline in the shared UI view.Where to look by owner:
lib/uiand the shared controls, the eval adapters and runner, the claim review step.libs/coreretry classification (an empty model reply is now retryable, refusals stay permanent),chat_trace, and anything in his multi-turn half that the builder touched.Testing
All 13 CI checks are green at the tip. Walked end to end yesterday on the 80-case default: 80 of 80 conversations driven, judge calibrated on the review sample, eval saved and run. The kiln_server side is deployed to staging for the same walk against a real server.
Not in this PR
🤖 Generated with Claude Code