[Amaury Levé (Evangelink)]

Pull request

Summary

Add an opt-in native Bazel build for TestFx and a separate CI experiment to measure build-cache and test-result reuse over real changes. The existing MSBuild/Arcade pipeline remains the required validation and shipping authority; this does not replace it or claim complete shipping parity.

• Compile the MSTest/MTP source graph directly with rules_dotnet, including modern framework variants, netstandard2.0/net462 compatibility, localization, analyzers/generators, and bounded Windows library/controller support. This is not an MSBuild wrapper; MSBuild is used separately for package consumers and the comparison baseline.
• Produce experimental native framework/adapter/analyzer/SDK packages and run the existing unit suites, portable samples, browser bundle, and benchmark paths. Preserve public-only development signing, friend identities, reference aliases, and source-path mapping.
• Add an independent Azure dogfood definition with cached, cold, forced-test, and matched MSBuild modes. It restores branch-scoped cache snapshots into fresh job directories, separates reused results from new executions, and refuses speedup claims when source, runtime, selection, outcomes, or timing evidence disagree.

Related issue: N/A; this is the user-requested staged Bazel experiment.

Design and decision record

Maintained research, alternatives, experiments, and limitations and commands/scope guide.

The default native toolchain remains SDK11, with the exact repository SDK12 available explicitly. The dogfood selects SDK12 and a net10.0 Release unit workload, preserving each suite's existing runtime layout. Cache transport initially uses Azure Cache@2; a remote action-cache service is deferred until transfer overhead justifies it. A parallel, non-required pipeline was chosen over a serial step in the existing build.

Important boundaries:

• managed_all and the compatibility/Windows aggregates primarily materialize assembly outputs; they are not equivalent to a full localized shipping build. Dogfood builds the selected test targets and their runtime dependencies instead.
• Public signing establishes development assembly identity, not official assembly/package signatures.
• Test caching is per test binary, not per MSTest method. A versioned, opt-in identity policy normalizes only the four clock fields in one existing DynamicData display name; original test bodies, raw TRXs, and other data remain unchanged.
• Native XAML/classic-UWP parity, the original net8 NativeAOT publisher path, complete acceptance/other-OS coverage, official signing, and release certification remain outside the delivered scope.

Demonstration

A text transcript is more useful than screenshots for this build/cache change. Observed locally during implementation:

Native SDK12 protocol run: 27 selected, 27 executed, 27 passed.
Fresh-output cache run:    27 selected, 0 newly executed, 27 reused.
                          453 build actions restored from the disk cache.
MSBuild protocol run:     The same 27 canonical identities and outcomes,
                          on the same runtime.

Cached TRXs are retained under reports/cached, never republished to Azure as new test executions. This demonstrates local reuse, not an Azure end-to-end speedup. Raw logs/binlogs/TRXs are retained locally and have not been uploaded as CI artifacts.

Validation and regression proof

Final helper source is committed in 8b4fd8b581d34b66c78f477f90b6817e9a58f7d8. The following standalone commands passed immediately before committing:

Exact command	Result
.\eng\bazel\test-dogfood-results.ps1	Exit 0; 69/69 standalone parser checks passed.
.\eng\bazel\test-dogfood-runner.ps1	Exit 0; 4/4 launcher/lifetime checks passed.
.\eng\bazel\test-dogfood-comparison.ps1	Exit 0; 5/5 comparison guards passed.
git diff --cached --check	Exit 0. Both new Azure YAML files also parsed successfully; this is not Azure service-side compilation.

An additional eight retained-report checks passed, giving 77 parser checks when those fixtures are included. The finalized parser/test file hashes were read back before commit.

The composed dogfood runs below were performed on implementation working states based on 0dcfbd1c2d9724e4891a042c357b3554d39cee30, not as a complete final-head rerun. Their records retain exact patch hashes. The two full runs share patch hash 902AE5FC99A2AABCBB55FEA18622B9FED44806BE28291DE2AF9AA47DF6341202.

For the commands below, the recorded path values were:

$e = Join-Path $PWD 'artifacts\bazel-dogfood-validation\20261104-104557-177b20c8'
$c = 'Q:\tfbc\dogfood-20261104-104557-177b20c8'

Exact effective command	Result
.\eng\bazel\dogfood.ps1 -Mode Cached -RunRoot (Join-Path $e 'native-hit') -CacheRoot $c -Targets '//:protocol_contract_tests'	Exit 0; 27 cached passes, 453 disk-cache build hits, fresh output root.
.\eng\bazel\dogfood.ps1 -Mode MSBuild -RunRoot (Join-Path $e 'msbuild-one3') -CacheRoot $c -Targets '//:protocol_contract_tests'	Exit 0; 27 executed passes, matching native identities/outcomes/runtime.
.\eng\bazel\dogfood.ps1 -Mode Cached -RunRoot (Join-Path $e 'native-full2') -CacheRoot $c	Exit 1, preserving child exit 3; all 13 suites selected 12,505 cases: 12,487 passed, 15 skipped, 3 failed.
.\eng\bazel\dogfood.ps1 -Mode MSBuild -RunRoot (Join-Path $e 'msbuild-full') -CacheRoot $c	Exit 1; 12,505 cases: 12,479 passed, 15 skipped, 11 failed. Nine failures exposed a missing isolated legacy-task fixture; the same two Win32 failures remained.
.\eng\bazel\dogfood.ps1 -Mode MSBuild -RunRoot (Join-Path $e 'msbuild-legacy') -CacheRoot $c -Targets '//:msbuild_tests'	Exit 0; after staging the existing netstandard2.0 sidecar contract, all 388 cases passed. No original tests were changed.

The native failures were two previously observed Win32-1346 impersonation failures and FailedLaunchDisposesProcessHandles observing 13 handles against an upper bound of 12. The handle case passed in the MSBuild run; no assertion, skip, or retry was changed to hide it. All 13 selections reconcile after the narrow clock-display projection.

Earlier producer/package validation at c910ab89ce7dc95c3ab2666ed62c4172a9950183 is recorded with exact commands and consumer provenance in the decision record: SDK/meta 40 steps passed, MSTest/runtime/satellite sensitivity 31 steps passed, analyzer package 15 cases passed, plus real Windows activation, browser exits 0/9, and portable samples. These are historical local results, not final-head CI certification.

Still unverified: registration/execution of the new Azure pipeline, real cross-agent Cache@2 restore/upload timings, a complete post-correction four-mode comparison, and the planned source/test/resource/configuration invalidation experiment. No percentage CI saving is claimed. The required pipelines remain unchanged and authoritative.

Test-value evidence

Requirement	Evidence
Missing test evidence cannot become green	BepRejectsMissingExpectedTarget: with the original 67-case suite unchanged, removing only the missing-target guard produced 66 passes and the intended failure (exit 1); restoring it produced 67 passes (exit 0).
Clock normalization is narrow and opt-in	TrxTimingPolicyDefaultsToExact, TrxTimingPolicyPreservesDurations, and NativeTrxTimingPolicyMatchesPairedSelections: the finalized 77-case parser suite passed; disabling only normalization in an isolated copy produced 70 passes and seven intended assertion failures; restoring it returned 77 passes. The real paired 3,380-case selections changed exactly one key each.
Cache/control accounting stays honest	MatchedControlsMeasureNetJobSavingsAndKeepReuseSeparate, EqualCountsDoNotHideDifferentTestIdentities, ForcedExecutionControlRejectsCachedTests, and IncompleteTimelineRemainsUnknownRatherThanZero passed in the five-case comparison suite.
Child invocation preserves the intended command and failure	ScriptPreservesColonBearingBazelOptions, WindowsPowerShellPreservesNamedBuildParameters, ProcessPreservesFailureExitAndBothLogs, and ProcessStreamsLogsBeforeExitAndJoinsTheChild passed.

The normalization counterfactual patch SHA-256 is 3fd457014af48ab69e8f196a1a87b3e9f69e7f4e463eb2a2355c59e2f6c603c2. Raw reports and original unit sources were preserved. Broader cache-invalidation sensitivity remains unproven, as called out above.

[github-actions]

🧪 Test quality review — PR #12204

Compared merge base 92e83edafb54811009aae479caf29a763f19a99a with head 03e3d3503573ab52dd7b97e2a6f6f3b9d929d21b. Evidence applies to this head, not every intermediate commit.

Test-value evidence

All evidence is static or unverified. This workflow runs no tests. The author's PR-body claims (for example the 67→66 and 77→70 counterfactual runs) concern the dogfood PowerShell parser suites, which are not in this PR's changed test files. I have no accessible paired execution record for them.

• Bazel build tools (resx/xlf/metadata/package version): BuildToolsTests asserts exact generated output (BOM, raw-string source, resource values, exit codes and error text) for valid and rejected input, and that no output file is written on rejection. Prediction: newline/BOM regressions or a dropped validation would likely fail. Static prediction only.
• SDK package generation: SdkPackageTests covers template/version preservation and rejection of contract drift. Static prediction only.
• Reference aliases: AliasTests covers alias identity, friend access and public key token. Static prediction only.
• EqtTraceLoggerContinuationTests: the isolated child host is now launched from Environment.ProcessPath or the test assembly instead of .dotnet/dotnet test. This is a behavior-preserving harness change. Both revisions would be expected to pass, but this is unverified here.
• ~100 Bazel/Starlark/PowerShell/CI/docs files: no tests were found for the .bzl rules or .ps1 scripts in this diff. These contracts are not assessed and remain unverified.

🧵 Parallel safety

STATUS: PARTIAL. I did not launch the specialist subtask. A direct check shows the new tests use per-test temp directories. EqtTraceLoggerContinuationTests launches a child host with a unique results directory. I performed no assembly-level parallelization analysis, and one config candidate was not analyzed. No findings come from the direct check, and this does not mean parallel safety is clean.

Per-test assessment

Pseudo-mutation evidence is not provided. Grading was a static review of the changed tests. Rows are grouped by test, with data-driven cases combined.

<table>
<thead>
<tr><th>Result</th><th>Quality</th><th>Test</th><th>Notes</th><th>How to improve</th></tr>
</thead>
<tbody>
<tr>
<td>Pass</td>
<td>A (90–100)</td>
<td>new <code>BuildToolsTests.<br>Resx_<br>PreservesTextAndEmitsBom</code></td>
<td>Exact output, BOM and values are asserted for LF and CRLF.</td>
<td>None</td>
</tr>
<tr>
<td>Pass</td>
<td>B (80–89)</td>
<td>new <code>BuildToolsTests.<br>Xlf_, Resx_, Metadata_*</code></td>
<td>Exit codes, error text and absent outputs are asserted.</td>
<td>None</td>
</tr>
<tr>
<td>Pass</td>
<td>B (80–89)</td>
<td>new <code>SdkPackageTests.<br>*</code></td>
<td>Template, dependency and drift-rejection contracts are asserted.</td>
<td>None</td>
</tr>
<tr>
<td>Pass</td>
<td>B (80–89)</td>
<td>new <code>AliasTests.<br>*</code></td>
<td>Alias identity, friend access and public key token are asserted.</td>
<td>None</td>
</tr>
<tr>
<td>Pass</td>
<td>B (80–89)</td>
<td>mod <code>EqtTraceLoggerContinuationTests.<br>TraceBridgePersistsEverySeverityInAnIsolatedHost</code></td>
<td>The child must exit 0 with an isolated results directory.</td>
<td>None</td>
</tr>
</tbody>
</table>

<sub>This advisory review is heuristic and non-blocking. No inline findings were published.

Re-run the combined review with /review-tests, or only the parallel-safety specialist with /parallel-audit.</sub>

🤖 Automated content by GitHub Copilot. Generated by the Test Quality Review on PR (on open / sync) workflow. · copilot · auto · ~34 AIC · ⌖ ~1 AIC · ⊞ 8K

[Copilot]

🔵 Needs a closer look

The workflow trigger omits Directory.Packages.props, allowing central Roslyn version changes to bypass dependency-alignment validation and merge with stale native Paket pins.

2 open issues

• Resolve the child results directory to an absolute path
• Run the dependency drift guard when central package versions change

_{Review effort: Balanced}

[github-actions]

🧪 Test quality review — PR #12204

Compared merge base 92e83edafb54811009aae479caf29a763f19a99a with head 3504d985dd3483c2bd25c4ddc440bb71f23ccc8c. Evidence applies to this head only.

Test-value evidence

Static, bounded pass only. No tests, builds, or mutations were run, and no paired execution records were available. Everything below is unverified / static prediction, not executed red-green proof.

• Bazel build/packaging infrastructure (about 100 new eng/bazel/**, .bzl, .ps1, docs, and workflow files): not assessed individually because of review budget. New unit tests cover only the C# build tools (Bazel.BuildTools.UnitTests, Bazel.SdkPackages.UnitTests) and the reference aliases (Bazel.ReferenceAliases.UnitTests). Starlark, PowerShell, and pipeline changes are unassessed.
• EqtTraceLoggerContinuationTests: the child test host is now launched directly from Environment.ProcessPath (or the assembly on .NET Framework) instead of dotnet test --project. Static prediction: a broken child launch or a non-zero exit would still fail child.ExitCode.Should().Be(0, ...). The exact-revision execution is unverified.
• Behavior-preserving or non-executable changes: not applicable.

🧵 Parallel safety

PARALLEL-SAFETY RESULT
STATUS: PARTIAL
NOTES: The parallel-safety specialist was not run because of review budget. The changed configuration candidate (BuildToolsTests.cs) and the new test assemblies were not assessed. This is not a clean result.
END PARALLEL-SAFETY RESULT

Per-test assessment

Per-test grading was not performed (review budget), so no grades are claimed. Changed test methods in scope, all Uncertain with no quality assigned:

<table>
<thead>
<tr><th>Result</th><th>Quality</th><th>Test</th><th>Notes</th><th>How to improve</th></tr>
</thead>
<tbody>
<tr><td>Uncertain</td><td>—</td><td>new <code>BuildToolsTests.<br>*</code> (6 methods)</td><td>Not graded; the tests exercise build tools.</td><td>Run the grading skill with the production tool sources.</td></tr>
<tr><td>Uncertain</td><td>—</td><td>new <code>SdkPackageTests.<br>*</code> (3 methods)</td><td>Not graded.</td><td>Run the grading skill with the production tool sources.</td></tr>
<tr><td>Uncertain</td><td>—</td><td>new <code>AliasTests.<br>*</code> (5 methods)</td><td>Not graded.</td><td>Run the grading skill with the alias fixtures.</td></tr>
<tr><td>Uncertain</td><td>—</td><td>mod <code>EqtTraceLoggerContinuationTests.<br>TraceBridgePersistsEverySeverityInAnIsolatedHost</code></td><td>Child launch now uses the host process directly.</td><td>Confirm the child run executes on both TFMs.</td></tr>
</tbody>
</table>

No inline findings were published; no verified actionable defect was identified.

_{This advisory review is heuristic and non-blocking. Re-run with /review-tests, or only the parallel-safety specialist with /parallel-audit.}

🤖 Automated content by GitHub Copilot. Generated by the Test Quality Review on PR (on open / sync) workflow. · copilot · auto · ~28 AIC · ⌖ ~1 AIC · ⊞ 8K

[Copilot]

🔵 Needs a closer look

Human assessment is needed to determine whether Cold-only cache uploads skew the whole-job timing comparison and overstate reported savings.

No open issues 🎉

2 resolved since last review

• Resolve the child results directory to an absolute path
• Run the dependency drift guard when central package versions change

_{Review effort: Balanced}

[github-actions]

🧪 Test quality review — PR #12204

Compared merge base 6876cf113dce6e1db55092779e73840697a0af73 with head 3431ca74f3196a39675874032557c08761429aa8. Evidence applies to this head, not every intermediate commit.

Test-value evidence

This is a bounded static review. Nothing was built or executed, and no paired red/green execution records were available. Every result below is a static prediction or unverified.

• Bazel build tooling tests (BuildToolsTests, SdkPackageTests, AliasTests): these are new tests for new tooling. They cover RESX/XLF handling, version templating, package-version selection, SDK package generation, drift rejection, and reference-alias identity. The names and visible assertions match the claimed contracts. Execution evidence is unverified, and the tests are not graded in detail.
• EqtTraceLoggerContinuationTests.TraceBridgePersistsEverySeverityInAnIsolatedHost (modified): the test now starts the child with the current host (Environment.ProcessPath or the assembly). Before, it ran dotnet test from .dotnet/. The decisive assertion is still child.ExitCode.Should().Be(0, ...), plus the existing persisted-severity checks in the child. The refactor is expected to pass before and after, but this is unverified.
• Non-test changes (.bzl, .ps1, MODULE.bazel, docs, pipelines, patches): not applicable or not assessed. This is a build-infrastructure change and I did not assess it in this run.

🧵 Parallel safety

PARALLEL-SAFETY RESULT: STATUS: PARTIAL. The specialist analysis was not completed in this run because of the size of the PR (115 files). This is not a clean result. Please run /parallel-audit.

Per-test assessment

Detailed per-test grading was not completed, because this PR adds about 600 lines of new tests plus a modified process-spawning test. The scorecard below therefore reports Uncertain for every group. It reflects missing evidence, not a defect. No inline findings were published. Existing review threads could not be checked for deduplication, so inline publication was withheld.

<table>
<thead>
<tr><th>Result</th><th>Quality</th><th>Test</th><th>Notes</th><th>How to improve</th></tr>
</thead>
<tbody>
<tr>
<td>Uncertain</td>
<td>—</td>
<td>new <code>BuildToolsTests.</code>, <code>SdkPackageTests.</code>, <code>AliasTests.*</code></td>
<td>Per-method grading was not completed in this bounded run.</td>
<td>Re-run <code>/review-tests</code> on a narrower scope for grades.</td>
</tr>
<tr>
<td>Uncertain</td>
<td>—</td>
<td>mod <code>EqtTraceLoggerContinuationTests.<br>TraceBridgePersistsEverySeverityInAnIsolatedHost</code></td>
<td>Child host selection was refactored; the exit-code assertion is kept.</td>
<td>Provide before/after execution results for this test.</td>
</tr>
</tbody>
</table>

<sub>This advisory review is heuristic and non-blocking.

Re-run the combined review with /review-tests, or only the parallel-safety specialist with /parallel-audit.</sub>

🤖 Automated content by GitHub Copilot. Generated by the Test Quality Review on PR (on open / sync) workflow. · copilot · auto · ~22 AIC · ⌖ ~1 AIC · ⊞ 8K

[Copilot]

🔵 Needs a closer look

The final revision lacks a complete Azure four-mode run validating the native toolchain, cross-agent cache reuse, and input invalidation, leaving key execution paths unverified.

No open issues 🎉

_{Review effort: Balanced}

[github-actions]

🧪 Test quality review — PR #12204

Compared merge base 6876cf113dce6e1db55092779e73840697a0af73 with head d45cf09562da4281be48831472cd5227117bef76. Evidence applies to this head, not every intermediate commit.

Test-value evidence

Static, partial assessment only. No tests were executed, and the PR's executed-evidence claims were not independently verified at this head.

• Bazel build/test infrastructure (115 files, ~21k lines): this is non-shipping, opt-in tooling. Only the new Bazel.BuildTools.UnitTests suite was examined in detail: the resx and xlf conversion cases and the invalid-catalog DataRow cases (culture, stale, duplicate, inline). They assert exit code, exact resource values, and that no output file is written. Static prediction: a regression in those behaviors would likely be caught. Unverified.
• EqtTraceLoggerContinuationTests: the child-host launch changed from dotnet test to a direct host invocation. The child.ExitCode.Should().Be(0, ...) assertion is retained. Static prediction: the isolation contract is still checked. Unverified.
• Not assessed: most .bzl/PowerShell/Azure pipeline files and the Bazel.SdkPackages and Bazel.ReferenceAliases tests (review budget). The PR body's red/green claims for the dogfood parser suites are author-reported and are not accepted here as paired execution proof.

🧵 Parallel safety

STATUS: PARTIAL. The parallel-safety specialist was not run in this pass, so there are no findings and no clean verdict. BuildToolsTests.cs was flagged as a parallelization candidate. It uses per-test Write/PathFor helpers with a TestContext constructor, which suggests per-test paths, but the helper bodies were not verified.

Per-test assessment

Pseudo-mutation evidence below consists of static predictions, not executed mutation results.

<table>
<thead>
<tr><th>Result</th><th>Quality</th><th>Test</th><th>Notes</th><th>How to improve</th></tr>
</thead>
<tbody>
<tr>
<td>Pass</td>
<td>A (90–100)</td>
<td>new <code>BuildToolsTests.<br>Resx_<br>PreservesTextAndEmitsBom</code></td>
<td>Exact values, BOM and generated source are asserted for LF and CRLF.</td>
<td>None</td>
</tr>
<tr>
<td>Pass</td>
<td>A (90–100)</td>
<td>new <code>BuildToolsTests.<br>Xlf_<br>AppliesTargetsAndPreservesMissingTargetFallback</code></td>
<td>Each translation state is asserted against its expected value.</td>
<td>None</td>
</tr>
<tr>
<td>Pass</td>
<td>A (90–100)</td>
<td>new <code>BuildToolsTests.<br>Xlf_<br>RejectsInvalidCatalog</code></td>
<td>Per-scenario error text, nonzero exit and no output file are asserted.</td>
<td>None</td>
</tr>
<tr>
<td>Pass</td>
<td>A (90–100)</td>
<td>new <code>BuildToolsTests.<br>Resx_<br>RejectsNonStringDataBeforeWritingOutputs</code></td>
<td>Error text and absence of both outputs are asserted.</td>
<td>None</td>
</tr>
<tr>
<td>Uncertain</td>
<td>—</td>
<td>new remaining tests in <code>BuildToolsTests</code>, <code>SdkPackageTests</code>, <code>AliasTests</code></td>
<td>Not graded in this pass because of review budget.</td>
<td>Re-run <code>/review-tests</code> for per-test grades.</td>
</tr>
<tr>
<td>Uncertain</td>
<td>—</td>
<td>mod <code>EqtTraceLoggerContinuationTests.<br>TraceBridgePersistsEverySeverityInAnIsolatedHost</code></td>
<td>The child-host command changed; production and child behavior were not traced.</td>
<td>Confirm the child run on net462 and on net10.0 under the dotnet host.</td>
</tr>
</tbody>
</table>

No inline comments were queued. Existing-thread and prior-review lookups were not completed, so duplicate suppression is unconfirmed.

<sub>This advisory review is heuristic and non-blocking.

Re-run the combined review with /review-tests, or only the parallel-safety specialist with /parallel-audit.</sub>

🤖 Automated content by GitHub Copilot. Generated by the Test Quality Review on PR (on open / sync) workflow. · copilot · auto · ~31 AIC · ⌖ ~1 AIC · ⊞ 8K

[Copilot]

🔵 Needs a closer look

Human assessment is needed because Cold-only action-cache uploads inflate its full job duration and may overstate the apparent savings from cached modes.

No open issues 🎉

_{Review effort: Balanced}

[Amaury Levé (Evangelink)]

/azp run microsoft.testfx

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[github-actions]

Final test-triage resolution — Azure Pipelines build 1666178

Result: inconclusive (no actionable test defect found; no issue warranted).

• The build (microsoft.testfx, 20261105.29) completed as succeeded. The timeline has no failed legs, crashes, hangs, or diagnostics. The only warning is low free memory on the "Test (unit tests only)" step.
• The collected reports contain 101,579 passed, 835 skipped and 6 "failed" records. All 6 come from Windows_App_Model_Diagnostics_Attempt1 fixture/probe outputs: intentional probes under extension-probes, NativeSidecar, and PackagedWinUI/Retries. Their messages are deliberate, e.g. "This probe deliberately fails the first retry attempt", "expected host failure", and "The first AppContainer host fails so Retry must activate a second host". They come from retry/native-sidecar scenario tests, not product regressions.
• No slowness regressions were found (0 of 30 slow candidates met the p95 and 60s floor criteria). No flakiness was proven (no CTRF fail-then-pass records).
• Caveat: evidence collection was incomplete (evidenceIncomplete=true, 1 artifact download failure), so this is not a claim that all legs are clean. Re-run or inspect the failed artifact download if a missing leg is a concern.

Confidence: high that the 6 failed records are not real failures; moderate overall because of the missing artifact.

🤖 Automated content by GitHub Copilot. Generated by the Pipeline Test Triage workflow. · copilot · auto · ~27 AIC · ⌖ ~1 AIC · ⊞ 10K