Raven x - #240
Open
TongLi31 wants to merge 16 commits into
Open
Conversation
…ONPATH Motivation: trajectory analysis of 177 raven failures on SWE-bench Verified found the dominant framework-attributable failure (47/60 cases) was the agent declaring completion after ad-hoc self-checks, because the bundled runtime's PYTHONPATH leaked into exec child processes and broke the workspace project's own test tooling (bundle packages shadowing a much older interpreter). - raven/__main__.py: drop raven's own install dir from PYTHONPATH at startup so exec children never inherit bundle packages; sys.path of the raven process itself is already resolved and unaffected. - raven/agent/loop/main.py: opt-in completion gate (env RAVEN_REQUIRE_REAL_TEST_EVIDENCE): a final text answer is only accepted after the session has run the project's own test runner at least once; bounded to 2 nudges so the gate itself can never loop. Evaluated on a 136-task SWE-bench Verified subset: 67 resolved vs 56 baseline (net +11); the gate fired in 13 tasks with no runaway loops. Co-authored-by: Claude (claude-fable-5) <noreply@anthropic.com>
Motivation: after wave-1 (environment isolation + completion gate), the remaining SWE-bench Verified failures cluster into generalizable methodology gaps, not benchmark specifics: over-wide edits that break neighboring behavior (10 of 34 remaining target failures show passed-before/failing-after tests), fixes that cover only the literal example and miss sibling code paths (16 of 34), and correct fixes reverted because an existing test encoded the old behavior. Add a compact, domain-general discipline block to the identity segment so every harness benefits (not only the SWE eval prompt): root-cause-first, requirement text over stale tests, smallest complete change applied to all sibling paths, verify with the project's own tooling, treat passed-before/failing-after as a self-introduced regression, review the full diff before declaring completion. Co-authored-by: Claude (claude-fable-5) <noreply@anthropic.com>
…licts A comprehension test by an independent agent (given only the block, no context) rated the first version 6/10 and found two defects: - the stale-test rule (Understand) and the regression rule (Verify) gave opposite default verdicts for the same observable event with no classification criterion; now the criterion is explicit (stale = asserts the exact old behavior the task asks to change; everything else is a self-introduced regression) and the two rules cross-reference each other. - 'smallest change' conflicted with 'fix ALL occurrences'; now scoped as smallest change per fix site, applied to every site sharing the flaw. Also restructured from flat compound bullets into Understand / Implement / Verify / Before-declaring-done phases with short imperative rules, matching the prompt style that stronger harnesses use. Co-authored-by: Claude (claude-fable-5) <noreply@anthropic.com>
…l result persistence Wave-4 completion-guardrail refinement, opt-in and layered on the wave-1 evidence gate (RAVEN_REQUIRE_REAL_TEST_EVIDENCE): - Evidence ledger in the agent loop: parse each real-test run's output as red/green (conservative regexes; unparseable output never triggers anything) and track non-doc file edits made after it. - RAVEN_GATE_STALE: when the model declares completion with untested edits, append ONE fact-stating reminder asking it to re-run or justify; doc-only edits exempt. - RAVEN_GATE_RED: when the model declares completion right after a failing run, append ONE inquiry with an explicit exit (a test asserting the old behavior the task changes is stale and may remain failing) so a correct judgment is never forced into iteration. - Persist oversized tool results as head 12k + tail 4k instead of head-only 16k: test runners put the pass/fail summary at the end, so the old truncation cut exactly the evidence later turns need. Both reminders fire at most once per turn, state only verifiable facts, and leave the decision to the model. Co-authored-by: Claude (claude-fable-5) <noreply@anthropic.com>
…ratch paths Three detector fixes surfaced by the 50-task variant exploration (6 of 12 v1-gate fires were false positives, one inducing a 163-turn re-verify loop): - Strip ANSI color sequences before matching test output: color codes glue onto adjacent digits (ESC[32m201 passed) and defeat word-boundary matching. - Recognize sympy's native runner invoked as a module (runtests without .py, sympy.doctest) and its output banners (test process starts, tests finished). - Do not mark evidence stale for edits under /tmp, /var/tmp, or /dev/shm: scratch files are not part of the deliverable (2 of 3 stale fires were /tmp reproducer scripts). - Gate nudge now says NOT to install packages and points at bundled runners (4 sessions responded to the nudge with pip install pytest, violating task constraints and adding skip/fail noise from pytest-8-on-old-repo). Co-authored-by: Claude (claude-fable-5) <noreply@anthropic.com>
…g self-written checks The old wording (a check you wrote yourself is NOT verification) is a dead-end instruction in projects that have no test suite. Reword as an evidence ranking: project tests first; if none cover the change, write one following project conventions and run it through a real runner; ad-hoc self-written checks are the weakest evidence because they re-encode the same assumptions as the change. The completion claim must state what evidence backs it.
Root-level change description for the raven-x branch: the runtime PYTHONPATH leak fix, the opt-in test-evidence guardrails (completion gate, stale-evidence reminder, red-test inquiry with a semantic exit), the engineering-discipline identity block, and the evaluation results backing each change. Written for reviewers (human or agent) landing on this branch. Co-authored-by: Claude (claude-fable-5) <noreply@anthropic.com>
Add a detached BackgroundJobRegistry (setsid, log file, survives agent exit) next to PTY exec sessions, and route background=true exec calls to it unconditionally, using a provided session name as the job name. Session-hosted background processes died with the agent and failed verifier-time checks in TB eval (4 tasks lost their servers). Also: - anchor shutdown/reboot/mkfs deny patterns to command position and report the matched pattern in guard errors - clamp over-limit exec timeouts (configurable tools.exec.maxTimeout) instead of rejecting them - teach session errors the creation path; suggest the matching tool on invalid parameters (registry did-you-mean) - register web_search only when an API key is configured - spill truncated exec/session output to ~/.raven/tool-output so it can be recovered with grep/read_file instead of re-running - log the final tool manifest (names + schema hash) for eval attribution Co-authored-by: Claude (claude-fable-5) <noreply@anthropic.com>
Every failure path should carry its own next step (G-1 of the tools improvement plan): - repair unambiguous case/format-mangled tool names (Read_File, execRead) and execute the intended tool with a note, instead of failing the turn - stop appending the generic change-approach suffix to validation errors and tool-authored errors; it buried the targeted guidance (did-you-mean, session creation path). Timeouts and unexpected exceptions keep it, as they have no better text - file tools name the recovery tool in not-found errors (list_dir / find to locate, write_file to create) Co-authored-by: Claude (claude-fable-5) <noreply@anthropic.com>
…kpoints Three long-turn robustness fixes (G-6 of the tools improvement plan): - refuse a byte-identical (tool, arguments) call after it has hard-failed 3 times running, instead of executing a fourth identical failure. Scoped to all-failed streaks only: identical successful repeats are legitimate polling (13 such streaks observed across passing eval tasks) and never trip it - repair dangling tool_call/result pairs in loaded history (synthetic aborted results, orphan results dropped) so a crash cannot poison later provider calls - persist the turn-so-far every 10 iterations. A turn is the unit of session persistence and benchmark tasks are one turn of hundreds of iterations; a crash mid-turn used to lose the whole trajectory Co-authored-by: Claude (claude-fable-5) <noreply@anthropic.com>
Add agents.defaults.profile (assistant | coding). The default keeps the personal-assistant identity byte-identical; "coding" renders a software-engineering identity following opencode's default prompt structure and content (tone, conventions, no-comments code style, verify-with-tests discipline, an explicit completion check, and an opencode-style <env> block), with every tool-specific passage rewritten for raven's tool surface: find/grep/read_file/edit_file/write_file routing over shell equivalents, background jobs for servers and over-ceiling work, sessions + exec_write/exec_read for interactive programs, and the untrusted-content rule. ContextBuilder._get_identity now delegates to the segment renderer so the estimator can no longer drift from the request path. Co-authored-by: Claude (claude-fable-5) <noreply@anthropic.com>
Attribution analysis of the tb21 full run (raven 39.3% vs terminus 44.9%), the P0/G fixes landed on this branch with their evidence, the 6-task re-verification results (pypi-server and configure-git-webserver flipped green; kv-store-grpc / install-windows-3.11 now fail on task logic instead of tool defects), and the deferred items (unified exec, todo/verification discipline) with measured regression exposure. Co-authored-by: Claude (claude-fable-5) <noreply@anthropic.com>
Combines the tools-improvement work (detached background jobs, deny pattern anchoring, timeout clamping, actionable errors, same-call breaker, mid-turn checkpoints, coding profile) with the raven-x completion-gate work. The SE-discipline block is hoisted into a shared constant rendered by BOTH identity profiles, so the eval harness's profile=coding selection keeps the discipline it was written for. Co-authored-by: Claude (claude-fable-5) <noreply@anthropic.com>
Resolves the registry boundary against main's ToolOutput/ToolResult split: the unwrap keeps main's display-text channel while retaining the tool-name-repair note and the no-generic-hint-on-tool-errors policy; the loop's test-evidence gate now reads the unwrapped model text. Co-authored-by: Claude (claude-fable-5) <noreply@anthropic.com>
feat(*): tools reliability fixes, coding profile, and main sync
vLLM 0.23.1 returns HTTP 200 with tool-call arguments that are not valid JSON — truncated, unescaped quotes, bare control characters, fragments of another tool-call format. Measured on a 500-task SWE-bench Verified run with Qwen3.6-35B-A3B: 0.17%-0.25% of calls. Raven-X repaired these with json_repair and then executed the result. That is worse than crashing: json_repair "completes" JSON cut off by the output limit into a syntactically valid object, so write_file wrote truncated content and silently lost the tail of the file, while the model believed it had succeeded and kept going. The run ends with a bad patch and no signal. Refuse to execute instead, and treat the two failure modes differently because their remedies are opposite: - Truncation (bracket-depth scan, with finish_reason == "length" as a supplementary signal) doubles the output budget and retries up to 4 times, capped at 32768, without consuming loop iterations. When the ladder is exhausted it still refuses, returning a tool result that says the call was cut off and asks for smaller edits. - Malformed JSON resamples the same request up to 3 times and adds nothing to history in the meantime — recording a bad shape teaches the model the shape is acceptable. Also unify the streaming and non-streaming paths on one parser. Streaming used a bare json.loads and, on failure, injected a `_raw_arguments` key that nothing in the tree consumes; it reached the tool through **params and surfaced as "missing required field", pointing the model away from the real cause. And fix an adjacent defect: chat_stream never fell back to self.generation the way chat_with_retry does, so the streaming output limit was the signature default of 4096 rather than the configured value. Tests: 12 cases over the parser and pure helpers, 5 over the loop side effects. Every bad sample is a real string captured from that 500-task run. Red/green verified: with the guard removed, the truncated write_file does write incomplete content. Full suite 4701 passed with 0 new failures (the 31 pre-existing failures in sandbox/channels/cli-theme tests reproduce identically with these changes stashed). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Type
Verification
Risk
Related Issues