fix(qa-synthesizer): read the schema router.ai() actually returns - #141
Open
hdimer wants to merge 2 commits into
Open
fix(qa-synthesizer): read the schema router.ai() actually returns#141hdimer wants to merge 2 commits into
hdimer wants to merge 2 commits into
Conversation
run_qa_synthesizer is the only reasoner that calls router.ai(); the other eighteen agents in this module use router.harness(), whose HarnessResult does carry a .parsed attribute. router.ai(..., schema=X) returns the validated X instance directly, so result.parsed raised AttributeError, the broad except swallowed it, and the function silently fell through to the tests_passed/review_approved heuristic every time. Read the schema instance directly and gate on its type, so a malformed tool-loop response still reaches the heuristic fallback deliberately rather than via a swallowed AttributeError. Fixes Agent-Field#113
Gating on isinstance meant a non-QASynthesisResult response (ai() returns a ToolCallResponse when the model content is not parseable JSON) reached the heuristic fallback while logging nothing. Issue Agent-Field#113 was diagnosed from that very note, so emit one deliberately instead of relying on a swallowed AttributeError to produce it. Adds a second test covering the path, and switches the router stub to the MagicMock/AsyncMock idiom the rest of the suite uses.
hdimer
marked this pull request as ready for review
August 23, 2026 08:38
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
run_qa_synthesizerreadresult.parsedoff the return ofrouter.ai(..., schema=QASynthesisResult). The SDK returns the validated schema instance directly, so that raisedAttributeError, the broadexcept Exceptionswallowed it, and the function fell through to thetests_passed/review_approvedheuristic on every call. The synthesizer's decision was never used. Fixes AgentField router.ai returns a QASynthesisResult directly, while SWE-AF expects an older wrapper with .parsed #113.run_qa_synthesizeris the one reasoner in the module that callsrouter.ai(); the eighteen other LLM-calling agents userouter.harness(), whoseHarnessResultreally does carry.parsed. They are correct and untouched.I used
isinstance(result, QASynthesisResult)rather than agetattr(result, "parsed", result)compatibility shim, becauseAgentAI.ai(..., schema=X)returnsschema(**json_data)in both0.1.113(this repo's floor pin) and current0.1.132, so there is no wrapper to fall back to on any supported version. The type gate also routes the one remaining non-schema return, aToolCallResponsewhen the model content is not parseable JSON, to the heuristic fallback deliberately and with its own note, rather than through a swallowedAttributeError. Happy to switch to the shim if you would rather keep the older shape working.Validation
make check- 1210 passed, 1 skipped (1208 passed before this change)Two tests in
tests/test_qa_synthesizer_direct_schema.py, both run against the unpatched file first to confirm they fail for the right reason. The first feeds inputs the heuristic resolves toapprovewhile the synthesizer returnsblock, so only the real decision can produceblock. The second fails if the non-schema path ever stops logging.Behavior Impact
Notes
Worth a look before merge, since this un-deadens two paths rather than just silencing an error:
actionis now the synthesizer's judgement instead of the heuristic's.blockwas previously reachable only when the reviewer setblocking; it can now come from the synthesizer alone, andcoding_loop.py:765turns that intoFAILED_UNRECOVERABLE.stuckfrom the synthesizer was alwaysFalse, socoding_loop.py:687only ever saw the_detect_stuck_loopbackstop at:809. The synthesizer's ownstucksignal becomes load-bearing here for the first time.Both are the behaviour of the code as written, but they are newly live, and by default they are driven by a
haiku-class model. If you would prefer to land the fix without switching those on in the same change, say so and I will split it.Written with AI assistance (Claude). I reviewed, tested, and verified the SDK behaviour myself.