Skip to content

fix(qa-synthesizer): read the schema router.ai() actually returns - #141

Open
hdimer wants to merge 2 commits into
Agent-Field:mainfrom
hdimer:fix/qa-synthesizer-direct-schema
Open

fix(qa-synthesizer): read the schema router.ai() actually returns#141
hdimer wants to merge 2 commits into
Agent-Field:mainfrom
hdimer:fix/qa-synthesizer-direct-schema

Conversation

@hdimer

@hdimer hdimer commented Aug 23, 2026

Copy link
Copy Markdown

Summary

  • run_qa_synthesizer read result.parsed off the return of router.ai(..., schema=QASynthesisResult). The SDK returns the validated schema instance directly, so that raised AttributeError, the broad except Exception swallowed it, and the function fell through to the tests_passed/review_approved heuristic on every call. The synthesizer's decision was never used. Fixes AgentField router.ai returns a QASynthesisResult directly, while SWE-AF expects an older wrapper with .parsed #113.
  • This is the only affected call site. run_qa_synthesizer is the one reasoner in the module that calls router.ai(); the eighteen other LLM-calling agents use router.harness(), whose HarnessResult really does carry .parsed. They are correct and untouched.

I used isinstance(result, QASynthesisResult) rather than a getattr(result, "parsed", result) compatibility shim, because AgentAI.ai(..., schema=X) returns schema(**json_data) in both 0.1.113 (this repo's floor pin) and current 0.1.132, so there is no wrapper to fall back to on any supported version. The type gate also routes the one remaining non-schema return, a ToolCallResponse when the model content is not parseable JSON, to the heuristic fallback deliberately and with its own note, rather than through a swallowed AttributeError. Happy to switch to the shim if you would rather keep the older shape working.

Validation

  • make check - 1210 passed, 1 skipped (1208 passed before this change)
  • Relevant manual test performed (if needed)

Two tests in tests/test_qa_synthesizer_direct_schema.py, both run against the unpatched file first to confirm they fail for the right reason. The first feeds inputs the heuristic resolves to approve while the synthesizer returns block, so only the real decision can produce block. The second fails if the non-schema path ever stops logging.

Behavior Impact

  • No behavior change
  • Backward compatible behavior change
  • Breaking change (explain below)

Notes

Worth a look before merge, since this un-deadens two paths rather than just silencing an error:

  • action is now the synthesizer's judgement instead of the heuristic's. block was previously reachable only when the reviewer set blocking; it can now come from the synthesizer alone, and coding_loop.py:765 turns that into FAILED_UNRECOVERABLE.
  • stuck from the synthesizer was always False, so coding_loop.py:687 only ever saw the _detect_stuck_loop backstop at :809. The synthesizer's own stuck signal becomes load-bearing here for the first time.

Both are the behaviour of the code as written, but they are newly live, and by default they are driven by a haiku-class model. If you would prefer to land the fix without switching those on in the same change, say so and I will split it.

Written with AI assistance (Claude). I reviewed, tested, and verified the SDK behaviour myself.

hdimer added 2 commits August 22, 2026 23:20
run_qa_synthesizer is the only reasoner that calls router.ai(); the other
eighteen agents in this module use router.harness(), whose HarnessResult
does carry a .parsed attribute. router.ai(..., schema=X) returns the
validated X instance directly, so result.parsed raised AttributeError,
the broad except swallowed it, and the function silently fell through to
the tests_passed/review_approved heuristic every time.

Read the schema instance directly and gate on its type, so a malformed
tool-loop response still reaches the heuristic fallback deliberately
rather than via a swallowed AttributeError.

Fixes Agent-Field#113
Gating on isinstance meant a non-QASynthesisResult response (ai() returns
a ToolCallResponse when the model content is not parseable JSON) reached
the heuristic fallback while logging nothing. Issue Agent-Field#113 was diagnosed
from that very note, so emit one deliberately instead of relying on a
swallowed AttributeError to produce it.

Adds a second test covering the path, and switches the router stub to
the MagicMock/AsyncMock idiom the rest of the suite uses.
@CLAassistant

CLAassistant commented Aug 23, 2026

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

@hdimer
hdimer marked this pull request as ready for review August 23, 2026 08:38
@hdimer
hdimer requested a review from AbirAbbas as a code owner August 23, 2026 08:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

AgentField router.ai returns a QASynthesisResult directly, while SWE-AF expects an older wrapper with .parsed

2 participants