Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
45 changes: 45 additions & 0 deletions .github/scripts/test_gh_aw_runtime_contracts.py
Original file line number Diff line number Diff line change
Expand Up @@ -345,6 +345,51 @@ def test_authored_test_review_footer_publishes_complete_advisory_review(self) ->
"/tmp/gh-aw/mcp-config/config.toml", "/tmp/gh-aw/pi-agent-dir/mcp.json",
}, {read["path"] for read in output["fsReads"]})

def test_authored_test_review_scorecard_publishes_as_markdown_table(self) -> None:
source, prompt = authored("shared/test-reviewer-shared.md")
table = prompt.split("Do not emit an empty table. Otherwise emit the table below. -->", 1)[1]
table = table.split("<!-- When it explains a finding", 1)[0].strip()
expected_table = """| Result | Quality | Test | Notes | How to improve |
| --- | --- | --- | --- | --- |
| Pass | B (80–89) | new `ClassName.Method_WhenSomething_ReturnsValue` | One exact assertion protects the claimed result. | None |
| Failed | C (70–79) | mod `ClassName.OtherMethod` | The test only checks non-null, not the claimed order total. | Assert Total is 10 for the arranged 50-unit order. |
| Uncertain | — | mod `ClassName.MissingMethod` | The requested method body could not be resolved. | Supply the method body and its referenced fixture. |"""
expected_body = (
expected_table + "\n\n> 🤖 **Automated content by GitHub Copilot.** Generated by the "
f"[{WORKFLOW_NAME}]({RUN_URL}/agentic_workflow) workflow. · \\~3 AIC\n\n"
f"<!-- gh-aw-agentic-workflow: {WORKFLOW_NAME}, run: {RUN_URL} -->"
)
for consumer in ("test-reviewer-on-pr.agent.md", "test-reviewer.agent.md"):
with self.subTest(consumer=consumer):
configuration, _ = authored(consumer)
self.assertIn("shared/test-reviewer-shared.md", configuration["imports"])
output = self.run_runtime(
"review", body=table,
footerMode=source["safe-outputs"]["submit-pull-request-review"]["footer"],
template=configuration["safe-outputs"]["messages"]["footer"],
)
requests = [call["params"] for call in output["calls"] if call["operation"] == "pulls.createReview"]
self.assertEqual([dict(owner="fixture", repo="workflows", pull_number=73,
commit_id=REVIEWED_SHA, event="COMMENT", body=expected_body)], requests)
self.assertIs(True, output["result"]["success"])
self.assertEqual([], output["errors"])

def test_review_table_preserves_escaped_cells_and_collapsed_rows(self) -> None:
body = r"""<details>
<summary>Additional tests</summary>

| Result | Quality | Test | Notes | How to improve |
| --- | --- | --- | --- | --- |
| Uncertain | — | mod `ClassName.Method_With_A_Long_Name_And_Generic_Input<Value>` | Compare `left \| right` with `<value> & "quotes"`. | Supply the missing fixture. |

</details>"""
output = self.run_runtime("review", body=body)
requests = [call["params"] for call in output["calls"] if call["operation"] == "pulls.createReview"]
self.assertEqual([dict(owner="fixture", repo="workflows", pull_number=73,
commit_id=REVIEWED_SHA, event="COMMENT", body=body)], requests)
self.assertIs(True, output["result"]["success"])
self.assertEqual([], output["errors"])

def test_staged_review_previews_without_publication(self) -> None:
output = self.run_runtime("review", body=REVIEW_BODY, staged=True)
self.assertEqual(dict(success=True, staged=True), output["result"])
Expand Down
48 changes: 48 additions & 0 deletions .github/workflows/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -93,6 +93,54 @@ inside the agent-visible tree. The agent reads this manifest after extraction.
Do not interpolate custom-step outputs into the prompt: gh-aw renders it before
those steps run. Missing inputs are incomplete analysis, not an empty scope.

#### Safe-output review tables

[The review on PR #12204](https://github.com/microsoft/testfx/pull/12204#pullrequestreview-5783835641)
displayed escaped `<table>`, `<code>`, and `<br>` markup instead of a scorecard.
The pinned gh-aw v1.0.1 report parser imports Markdown pipe tables as typed
tables; raw HTML table markup is literal text, not an allowed table element.
Both reviewers therefore use Markdown pipe tables, including evidence and
parallel-safety tables, without changing the renderer or weakening sanitization.

Keep each row on one physical line, escape literal pipes as `\|` even in inline
code, and leave blank lines around tables, including inside `<details>`.
Use backticks for complete test names instead of HTML spans or forced breaks.
Empty grading scopes still omit the table.

`test_authored_test_review_scorecard_publishes_as_markdown_table` extracts the
shared prompt's actual scorecard example and checks the final GitHub review
request through the unchanged pinned parser, renderer, and review buffer for
both consumers. `test_review_table_preserves_escaped_cells_and_collapsed_rows`
also checks long generic names, literal pipes, quotes, and `<details>` overflow.
The existing offline runtime suite runs both without posting a review. Set
`$actions` to the unmodified scripts installed by the pinned setup action and
`$baseline` to an isolated archive's `.github\workflows` directory:

```powershell
$cases = @(
'GhAwRuntimeContractTests.test_authored_test_review_scorecard_publishes_as_markdown_table'
'GhAwRuntimeContractTests.test_review_table_preserves_escaped_cells_and_collapsed_rows'
)
python .github\scripts\test_gh_aw_runtime_contracts.py --actions-dir $actions --workflow-root $baseline @cases -v
python .github\scripts\test_gh_aw_runtime_contracts.py --actions-dir $actions @cases -v
```

Local evidence (November 5, 2026) retains these same tests against base
`6876cf113dce6e1db55092779e73840697a0af73`: the scorecard case fails for both
consumers because the final request starts with `&lt;table&gt;`, while the
escaped-cell case passes (two methods executed, exit 1). The corrected selection
passes both methods with no skips (exit 0), and the complete runtime suite
passes all 15 methods. Strict compilation with ShellCheck passes both consumers
with zero warnings; only their generated body hashes change.

The GitHub Markdown API rendered the actual emitted fixture payload into two
five-column tables. A local-only browser preview in the original review layout
(1280 × 721 viewport, 800- and 480-pixel review widths) preserved test/result
associations and escaped text, with working collapsed/expanded overflow and no
overlapping cells. Narrow columns wrap heavily at 480 pixels. This is rendering
and component evidence, not a live model run or review publication; neither
workflow was dispatched and the existing review was not edited.

#### Typed safe-output footer templates

[Run 41640057298](https://github.com/microsoft/testfx/actions/runs/41640057298/job/125933099017?pr=12185)
Expand Down
57 changes: 21 additions & 36 deletions .github/workflows/shared/test-reviewer-shared.md
Original file line number Diff line number Diff line change
Expand Up @@ -880,7 +880,15 @@ comment before the parallel-safety result has been considered.

Use **exactly one** `submit-pull-request-review` call with
`event: "COMMENT"`. The inline comments published in this step are bundled into this
review. Structure the body as follows:
review.

Use **GitHub-flavored Markdown pipe tables** for every table in the review,
including test-value evidence, parallel-safety assemblies, and the per-test
scorecard. The pinned safe-output renderer recognizes Markdown tables but
escapes raw HTML table markup. Do not use `<table>`, HTML `<code>` spans, or
`<br>` tags, and do not wrap a published table in a code fence.

Structure the body as follows:
Replace `<pr_number>`, `<merge_base>`, and `<head_sha>` below with the exact
manifest values; never publish the placeholders.

Expand Down Expand Up @@ -931,34 +939,11 @@ Pseudo-mutation evidence below consists of static predictions, not executed muta
No new or modified test methods were identified in the changed regions.
Do not emit an empty table. Otherwise emit the table below. -->

<table>
<thead>
<tr><th>Result</th><th>Quality</th><th>Test</th><th>Notes</th><th>How to improve</th></tr>
</thead>
<tbody>
<tr>
<td>Pass</td>
<td>B (80–89)</td>
<td>new <code>ClassName.<br>Method_<br>WhenSomething_<br>ReturnsValue</code></td>
<td>One exact assertion protects the claimed result.</td>
<td>None</td>
</tr>
<tr>
<td>Failed</td>
<td>C (70–79)</td>
<td>mod <code>ClassName.<br>OtherMethod</code></td>
<td>The test only checks non-null, not the claimed order total.</td>
<td>Assert Total is 10 for the arranged 50-unit order.</td>
</tr>
<tr>
<td>Uncertain</td>
<td>—</td>
<td>mod <code>ClassName.<br>MissingMethod</code></td>
<td>The requested method body could not be resolved.</td>
<td>Supply the method body and its referenced fixture.</td>
</tr>
</tbody>
</table>
| Result | Quality | Test | Notes | How to improve |
| --- | --- | --- | --- | --- |
| Pass | B (80–89) | new `ClassName.Method_WhenSomething_ReturnsValue` | One exact assertion protects the claimed result. | None |
| Failed | C (70–79) | mod `ClassName.OtherMethod` | The test only checks non-null, not the claimed order total. | Assert Total is 10 for the arranged 50-unit order. |
| Uncertain | — | mod `ClassName.MissingMethod` | The requested method body could not be resolved. | Supply the method body and its referenced fixture. |

<!-- When it explains a finding or detail was requested, add a compact
Pseudo-mutation evidence block keyed by test: exact change, witness,
Expand Down Expand Up @@ -999,15 +984,15 @@ Per-test table rules:
F → D → C → B → A, then fully-qualified name. Unresolved quality is `—`.
- If there are more than 50 rows, show Failed and Uncertain rows first, sample
Pass rows, and collapse overflow into `<details>`.
- Use raw HTML `<table>`, not a Markdown pipe table. The sanitizer allows
`table`, `thead`, `tbody`, `tr`, `th`, `td`, `code`, `span`, `sub`, `sup`,
`br`, `details`, and `summary`. Do not use `<colgroup>`, `<col>`, `<wbr>`, or
invisible Unicode wrap characters.
- Keep each row on one physical line with the same five columns as the header.
Escape literal `|` characters inside cells as `\|`, including inside inline
code. Serialize the completed Markdown body normally in the safe-output JSON.
Leave a blank line before and after each table, including inside `<details>`.
- Render quality and score band together, for example `B (80–89)`; use `—`
for unresolved tests and never emit a fake-precise numeric score.
- In the Test column, prefix `new` or `mod` outside the `<code>` span when known,
drop the namespace, and insert `<br>` after the class separator and each `_`
so long MSTest names wrap without changing their copyable text.
- In the Test column, prefix `new` or `mod` outside the Markdown code span when
known and drop the namespace. Keep the complete `ClassName.MethodName` in
backticks without HTML tags or invisible Unicode wrap characters.
- Keep Notes and How to improve separate. Actions follow the result, not the
letter grade: Failed requires an evidence-backed correction, Pass uses
`None`, and Uncertain names the evidence needed.
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/test-reviewer-on-pr.agent.lock.yml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion .github/workflows/test-reviewer.agent.lock.yml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading