Conversation
Contributor
|
Important Draft PR not reviewedDraft PRs are not automatically reviewed by default.
To automatically review draft PRs, update your CodeRabbit configuration: reviews:
auto_review:
drafts: trueThanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
hubgan
force-pushed
the
feat/flow-step-failure-messages
branch
from
September 18, 2026 11:59
01a9d59 to
27b9a79
Compare
…f dropping them A text check that failed only on a line break or a tab printed two identical `expected:`/`actual:` lines, in the CLI and in the MCP result alike: both renderers replaced every control character with a space, and those lines are the only place the found text is printed. The difference survived in `--json` alone. Escape the characters instead. Each value still renders on one line and no raw escape sequence reaches the terminal.
…sted step A `tool: flow-execute` step builds its one-line failure from the inner step's reason. A failed check now keeps the text it found, and the advice about it, beside that reason, so the outer step said "did not equal" with no found text and "no element matched" with no scroll-to advice. Nothing else in a composed run prints the inner step, and the recorder writes this step form whenever the target is not a resolvable sibling flow. Carry `expected`, `actual`, `hint` and `indeterminate` from that inner step onto the outer one, where the CLI and the MCP result already print them.
… the app An app that crashed, was terminated, or emptied its own screen after the element was seen reads here exactly like a tree source that stopped answering: `isBlindRead` cannot tell them apart. The shared hint still promised "this is not a verdict on the app; re-run", so a real crash or a blank-screen regression was labelled environment noise, and the docs repeated the claim. Order the checks instead, and put the app first.
A failed snapshot rounded the measured diff to two decimals for display but compared it at full precision, so the step's own expected: and actual: lines could say the value found satisfied the value wanted: a 0.0028% diff against `maxMismatch: 0` failed as `diff 0.00% > 0%` with `expected: ≤ 0%` and `actual: 0.00%`. Every tolerance below 0.005% reads that way, and so does a real mismatch that rounds down onto its tolerance. The measured value now takes as many decimals as it needs to stay on the side of the tolerance the comparison put it on.
…teral
For `text: { matches: … }` the `expected:` line quoted the pattern as a string
literal and doubled every backslash, so the step line one row above printed
`matches /^Taps: \d\d\d$/` while the line the feature exists for printed
`expected: "^Taps: \\d\\d\\d$"` — a valid-looking pattern that matches a
literal backslash followed by `d` when it is copied back into the flow.
Nothing on the wire said the value was a pattern. A step report now carries
`expectedKind: "pattern"` beside `expected`, and both renderers print such a
value in slash delimiters, which is the spelling `describeTextExpectation`
already uses for a pattern in the step line and the reason. `expected` keeps
the raw pattern source, so a `--json` consumer reads the same value as before.
A tree source can answer with an empty tree and its own "I could not see the app" flags — an unattached Vega automation toolkit, or an AX service asking to be relaunched. `assert` and `idle` already refuse to judge such a read. The steps that resolve a frame reported `no element matched selector X` and advised adding a `scroll-to` step, so the one step that committed to a verdict committed to the wrong one: nothing was ever looked at, scrolling cannot help, and editing the flow is the wrong move. `settleTree` discarded the reader's flags when it narrowed its read to a tree, so `waitForFrames` could not tell a blind read from an empty screen. It now returns the whole read, the miss carries the blind-read flags, and `selectorMiss` reports such a miss as indeterminate with the reader's own repair. A genuinely empty screen still reports the miss and the scroll-to hint.
The failure-message skills told the agent to try the `hint:` line first, without qualification. Most of the failure surface emits no hint, including the commonest failures: `assert`/`await` with `exists`, `visible` or `hidden`, `launch:`, `script:`, `run:` and every `tool:` step. The agent reading those sentences looks for a line that is not there.
The recap that #1184 prints after a directory run, and above the verdict of a single run, shows the reason of the first failed step. That reason used to carry the text the step found and the advice: `its text was "Total $41.50" (wanted to equal "$42.00")`, `… — if it is off-screen, add a scroll-to step`. This branch moves both into the `expected:`, `actual:` and `hint:` lines, so on top of #1184 the recap printed `its text did not equal "$42.00"` and nothing else: the section meant to be read at the end of a long log no longer said what the step found. The recap now prints those lines under the reason, formatted and escaped as under the step itself.
hubgan
force-pushed
the
feat/flow-step-failure-messages
branch
from
September 21, 2026 13:44
27b9a79 to
c146531
Compare
A tree read refused with a validation failure (an Apple system app, or an app that left the foreground) is refused again on every run. The step still could not check the screen, so a when guard keeps erroring on it, but the step report no longer gets indeterminate: true or the shared re-run hint. Its reason already says what to change in the flow.
The own-text hint put the device text in raw quotes, and the renderers then doubled its backslashes, so 'Say "hi" C:\x' printed as neither raw text nor JSON. The tool-server now JSON-quotes the own text, the CLI and MCP escape only control characters in a hint and a snapshot value, and the recorder warning JSON-quotes the actual text.
The detail lines escaped only C0 controls, so a found text that differed from the expected one by a no-break space, a U+202F, a zero-width or bidi character, DEL or a C1 control still printed identically. The CLI and MCP now escape all of them, in quoted values, hints, snapshot values and patterns.
…step The outer tool step printed an inner snapshot's tolerance and diff as quoted text (expected: "≤ 0.5%"), because the renderers leave a value unquoted only on a snapshot step. The nested step now takes only the inner snapshot's hint. Its reason already quotes the inner reason, which holds both values.
7 of 1000 pixels computes 0.7000000000000001%, which fails maxMismatch: 0.7. Every fixed rounding prints 0.7, and the exponential fallback printed 7.00e-1%, so the step read 'diff 7.00e-1% > 0.7%'. The fallback now prints the shortest exact spelling, which stays on its side of the tolerance.
A determinate assert or await whose final poll could not read the UI tree put 'the final poll could not read the UI tree: ...' in hint:. That note describes the read, not what to try, and on iOS it can be a whole multi-sentence error. It closes the reason again, as before the hint existed, and the recorder warning still ends with it.
The recorder caps its quoted verdict at 200 characters. The own-text hint adds about 70 characters of fixed wording that the old one-sentence verdict did not have, so a found text that used to fit was cut out of the middle. The cap now includes that wording.
…repair When every read came back empty with the reader's own 'I could not see the app' flags, assert, await and when got the shared re-run hint, while tap and idle passed on the reader's repair (on Vega: relaunch the foreground app so the toolkit attaches). They now pass it on too, and fall back to the shared hint only when the reader gave none.
…the screen A cropOn (or tap-family) miss took its blind verdict from the last settled read alone. After rounds that really read the screen and did not find the element, one blind final read still reported 'was never looked for' with the reader's relaunch hint. The miss is now judged on the last read that saw the screen, and is blind only when every read in the window was.
Vega's tree keeps an off-screen node at zero area, so a cropOn that finds only such a node reaches the zero-area hint, which said to add a scroll-to step. Vega refuses scroll-to as a touch directive. On Vega the hint now names the causes without that advice.
A tap-family miss or an idle wait on an empty, reader-flagged read said 'not the app rendering nothing'. When the reader gave no hint, the shared hint under it said a screen the app emptied itself reads the same, and on a physical iPhone, whose reader flags every childless tree, the claim was false. The reason now states only that the screen was never read.
indeterminate: true reached only --json and --json-stream, so a reader of the CLI output or of the flow-execute MCP result could spot a check that never ran only from the prose of its reason. Both now print an 'indeterminate: the check did not run' line above the hint.
A swipe whose one end resolved and whose other end missed had no test, so counting the zero-area matches of the wrong end went unnoticed. The new cases put different match counts on the two ends, and the endpoint-miss test now checks the exact reason and hint.
…mpt cap Neither the 'not found after N scroll attempts' reason nor its hint was asserted. A tree that changes on every scroll and never shows the target now runs scroll-to to its cap.
One test paired a text mismatch with indeterminate: true, and another a diff failure with a size-drift hint; no step reports either pair. They are split into real shapes, and new tests cover a flow-execute nested in a flow-execute, built from a real middle run.
The flow-visual tests call runSnapshot directly, so nothing checked that a run's snapshot step report carries its hint, expected and actual. Runs with a missing baseline, a diff over maxMismatch and a cropOn size change now do.
…sted No directory-run test checked the expected, actual, indeterminate and hint lines in both the per-flow block and the Failed flows recap, and alignment was checked only at depth 0 with step 3. Both are now covered, including depth 1 and 2 and steps 100 and 1000.
Where the detail block sits beside a tool step's result was untested, and no test rendered a report the tool-server really produced. A failed nested flow-execute step and a verbatim report from a branch run now cover both.
The skills said to try the hint first, before a screenshot and describe, so an agent added a scroll-to step for a selector miss before it looked at the screen. They also called every unreadable tree an environment error, left out indeterminate steps, and said that no tool: step and no exists, visible or hidden check has a hint. They now list the lines, say to look at the screen before a hint, and say to look at the app first when a check did not run.
The reference said a hint is the first thing to try and never restates the reason, that only control characters are escaped, that a pattern can be copied back into matches: with its slashes, and that indeterminate marks every unreadable tree and always means run again. It now matches the code, names the flow-execute result and the indeterminate line, and says which steps can have the flag. The features page and the CLI help name the detail lines in the Failed flows recap.
Several comments still described the code before this PR's changes: the StepReport.hint doc said a hint never restates its reason, the indeterminate doc left out the frame-resolving steps, the tier-3 note said nothing about an empty final read, FrameMiss said matched counts every alternative, the renderer docs left out the detail lines and said they check the status, the pattern comments said the slashes copy back into matches:, and a physical iPhone blind-read test still described the old settleTree.
…hint" This reverts commit bc1e241. The finding it fixed cannot occur. No flow tree source sends an empty, flagged read without a hint: only Vega's reader flags an empty read, and it always adds its relaunch hint. A physical iPhone's flow source throws on that shape instead. So the shared hint never sits under the blind reason, and on Vega the claim 'not the app rendering nothing' holds.
No test combined the own-text hint with a failed final poll, or capped a long own text in the hint. The blind tap and type tests put Vega's flags on a native-devtools read, which no source sends, and Vega refuses a tap before it reads a tree. They now resolve a Vega cropOn frame, the one path a flow meets that read on, and each blind test names the source that sends its flags.
…e cuts actual stopped at 300 characters with an ellipsis inside the quotes, so a difference late in a large container's text was lost from the report and the cut read as device text. The report now keeps the whole text, as the old reason did. The CLI and MCP print its first 300 characters and then '… (N more characters)' outside the quotes, and the own-text hint does the same.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
When a flow step fails, the step report gives one text, the
reason. This text has these problems:tapon an element that is in the UI tree, but has no area on the screen, gives the same message as atapon an element that does not exist.What
reasonof a failed step states what failed. The advice that the old reasons carried moves tohint. A few reasons still carry advice: the snapshot reason for a different app, theflow-executeprerequisite reason, andpinch/rotateon a physical iPhone.expected: the value that the check wanted.actual: the value that the check found, in full. The CLI and MCP lines show the first 300 characters of a text and then the number of characters they do not show, outside the quotes.hint: a thing to try first, or a fact that helps find the cause, for example the element's own text.indeterminate: truewhen it did not do its check, because Argent could not read the UI tree:await,assertoridlestep, or awhenguard, whose reads failed, or came back empty and degraded;tap-family step or asnapshotwithcropOnwhose every read came back empty and degraded (only Vega's reader does this);tool: flow-executestep whose failed inner step had the flag.validationfailure, for example a flow that reads an Apple system app) gets no flag and no shared hint. Its reason already says what to change.flow-executeMCP result showexpected,actual,indeterminateandhintas lines below the step.Failed flowssection of a directory run (feat(cli): show the failed flows and a re-run command at the end of a directory run #1184), and the repeat of the failed step above the result of a single run, also show these lines below the reason.tapon an element with no area on the screen now tells you that the element is in the tree. The hint gives the possible causes.--jsonand--json-streamshow the new fields. The other fields do not change. A step without the new fields is the same as before.These steps get the new fields:
tap,long-press,swipe,type,pinch,rotateorcropOnfinds no elementhintassertorawaitwithtextfinds a different textexpected,actual, andhintwhen the element's own text is differentassert,await,idleor awhenguard cannot read the UI treehint,indeterminatetap-family step orcropOnon Vega whose every read is empty and degradedhint,indeterminatetool: flow-executewhose composed flow failedexpected/actualscroll-todoes not find the targethintsnapshothas no baselinehintsnapshothas a different image sizeexpected,actual, andhintforcropOnsnapshotdifference is more thanmaxMismatchexpected,actualReason texts that changed
If you search logs for an old reason text, use this list. Advice that this PR removes from a reason is now in
hint.no visible element matched selector X — if it is off-screen, add a scroll-to step before this oneno element matched selector X, orN element(s) matched X but none was visible (zero-area frame)element matched X but its text was "T" (own text "O") (wanted to equal "E")element matched X but its text did not equal "E"(alsocontainandmatch)no baseline for "N" on this device class — expected P, nothing was compared. Run with updateBaselines …no baseline for "N" on this device class — expected P, nothing was comparedbaseline is WxH but the cropOn region is WxH (K) — nothing was compared. The element's size drifted …baseline is WxH but the cropOn region is WxH (K) — nothing was comparedcould not read the UI tree while waiting for the screen to settle — check the app … Underlying error: Ecould not read the UI tree while waiting for the screen to settle: Ethe tree source never answered within the step's Nms — raise this step's timeout: …the tree source never answered within the step's Nmsthe UI tree source answered and then stopped: … for the rest of the wait — check the app …the UI tree source answered and then stopped: … for the rest of the waitthe UI tree read back empty and degraded while waiting … not the app rendering nothing. <source advice>the UI tree read back empty and degraded while waiting … not the app rendering nothingno visible element matched selector X — if it is off-screen, …(VegacropOn, every read blind)the UI tree read back empty and degraded, so X was never looked for — this is the reader reporting it could not see the app, not the app rendering nothingdiff 0.00% > 0%diff 0.002% > 0%: more decimals when two decimals would put the value on the wrong side ofmaxMismatch, and the full value when only float error separates them. This also changes the reason of a passing snapshot whose rounding crossed the toleranceHow
hint,expectedandactual. The runner copies them to the step report.indeterminateand the shared hint in one location. A step with its own hint keeps it. A read refused with avalidationfailure gets neither.flow-add-step) puts the reason, the actual text and the hint into its warning. Thus its warning keeps the data that the old reason had.renderStepDetailLines. The live output, the full report and the directory run use it. TheFailed flowssection uses the same text of the lines, with the indent of that section.\nor\u00a0: control and format characters, and each space other than U+0020. Thus a value cannot add a line to the output, and a value that differs only by such a character looks different. A quoted value, and device text that a hint quotes, use JSON escapes.Before and after
I ran the CLI of this branch on a private headless Chrome. The "before" CLI is #1183. I removed the paths, and I made the long text shorter.
Before:
After:
For this run, I stopped the Chrome during an
awaitstep. Before, the step showed only the reason. After:In
--json-stream, the same step has these fields (reasonremoved):{ "index": 0, "kind": "await", "flow": "l13_await", "target": "visible \"Never there\"", "status": "fail", "hint": "check the app first — a crash, or a screen the app emptied itself, reads the same here as a tree source that stopped answering — then check the device and the tree source; re-run before you edit the flow", "indeterminate": true, "durationMs": 8006 }A flow that reads an Apple system app is refused on every run. Its step keeps the reason only, with no flag and no re-run hint (iOS 26.5 simulator):
The
flow-executeMCP result, before:After:
The
Failed flowssection at the end of a directory run. Before (main, with #1184), the reason contained the found text and the advice:After:
Tests
indeterminateon anawaitstep and on awhenguard;--json. The flows fail a tap, a text check, a long text check, anawait, ascroll-toand two snapshot steps, and one flow file is not valid.--jsonfor one of them.awaitstep and awhenguard. I stopped Chrome during each, and each showed the shared hint. For theawaitstep,--json-streamshowedindeterminate: true.flow-executeMCP tool, through the MCP server of this branch.re-run:command from theFailed flowssection, and a flow with a nestedflow-executestep.assertand awhenguard on an Apple system app (no flag, no hint), and a flow that leaves the app through Safari (the final-poll note closes the reason).assertandawaitwith the toolkit not attached (the toolkit's own hint), and acropOnmiss when the device stops in the middle of the wait (the verdict of the reads that saw the screen).flow-executeMCP: an own text with quotes and backslashes, a zero-width space, a nestedflow-executeof a failed snapshot, a snapshot diff of 7 of 1000 pixels againstmaxMismatch: 0.7, and anawaitwhose Chrome stopped (theindeterminate:line).~/.config/opencode/opencode.jsonc. This PR does not change the installer.Not in this PR
(run cancelled)after Ctrl-C. Ctrl-C stops the CLI before a report comes. The tool-server stops a run only when the client closes the connection. Thus no client can show a report withaborted.Docs
This PR updates
reference/flow-yaml.mdxandfeatures/flows.mdx. It also updates theargent-create-flowandargent-qa-flowsskills: an agent records the detail lines, looks at the screen before it uses a hint, and looks at the app first when a check did not run.