Add dotnet test integration tests for all Apple platforms#25320
Add dotnet test integration tests for all Apple platforms#25320jonathanpeppers wants to merge 12 commits into
dotnet test integration tests for all Apple platforms#25320Conversation
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
jonathanpeppers
force-pushed
the
jonathanpeppers/add-mlaunch-mtp-tests
branch
3 times, most recently
from
5c37575 to
4c9418b
Compare
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
✅ [CI Build #3885b03] Build passed (Build macOS tests) ✅Pipeline on Agent |
This comment has been minimized.
This comment has been minimized.
Create DotNetTestTest that runs 'dotnet test' on projects created from the iostest, tvostest, maccatalysttest, and macostest templates. This verifies the MTP (Microsoft Testing Platform) pipeline works end-to-end, including the --server/--dotnet-test-pipe argument passthrough via RunArguments. Also add 'test' as a supported verb in DotNet.Execute. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
macOS and Mac Catalyst don't use mlaunch (they use 'open' via Desktop.targets), so MTP support requires a different approach. Comment out those test cases for now. Also skip the /v:diag and /consoleloggerparameters workaround args when the verb is 'test' to prevent them leaking through to the MTP test runner via RunArguments. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Update mlaunch NuGet package to include MTP (Microsoft Testing Platform) support for iOS and tvOS simulator test execution. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The dotnet test MTP flow calls ComputeRunArguments via the MSBuild API with loggers: null, making errors invisible. Split into three steps: 1. dotnet build - ensures app bundle is created 2. dotnet build /t:ComputeRunArguments - verifies targets work with full diagnostic output 3. dotnet test --no-build - runs the actual test Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The standalone ComputeRunArguments target fails on CI because GetAvailableDevices filters out all simulators. Since dotnet test handles ComputeRunArguments internally, just use build + test --no-build. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
dotnet test MTP internally calls ComputeRunArguments via MSBuild API without forwarding /p: properties from the command line. This caused ComputeRunArguments to fail because the TargetPlatformVersion did not match the installed SDK workload. Setting the property in the project file ensures it is available during all MSBuild evaluations. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
dotnet test MTP flow doesn't forward /p: properties or produce binlogs for its internal ComputeRunArguments call. Using Execution.RunAsync directly avoids these issues and includes the full output in the assertion message for better CI diagnostics. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The dotnet test MTP flow uses MSBuild API internally for ComputeRunArguments but doesn't load the workload SDK resolver. With --no-build, this means the iOS SDK can't be resolved at all. Without --no-build, dotnet test does a full build+run cycle where the workload resolver is properly initialized. Also use Execution.RunAsync directly instead of DotNet.Execute to avoid /p: and /bl: args that dotnet test doesn't forward. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The NUnit runner process sets MSBuildSDKsPath and MSBUILD_EXE_PATH which interfere with the child dotnet process SDK resolution. Every DotNet.Execute call clears these; we must do the same. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The ComputeRunArguments target requires a booted simulator to select a device. Query available devices via ComputeAvailableDevices target and boot one before running dotnet test. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
jonathanpeppers
force-pushed
the
jonathanpeppers/add-mlaunch-mtp-tests
branch
from
3885b03 to
f702ffe
Compare
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
🔥 [CI Build #f702ffe] Build failed (Detect API changes) 🔥Build failed for the job 'Detect API changes' (with job status 'Failed') Pipeline on Agent |
🔥 [PR Build #f702ffe] Build failed (Build packages) 🔥Build failed for the job 'Build packages' (with job status 'Failed') Pipeline on Agent |
This comment has been minimized.
This comment has been minimized.
|
🔥 Unable to find the contents for the comment: D:\a\1\s\change-detection\results\gh-comment.md does not exist :fire Pipeline on Agent |
🔥 [CI Build #f702ffe] Test results 🔥Test results❌ Tests failed on VSTS: test results 27 tests crashed, 0 tests failed, 0 tests passed. Failures❌ cecil tests🔥 Failed catastrophically on VSTS: test results - cecil (no summary found). Html Report (VSDrops) Download ❌ dotnettests tests (iOS)🔥 Failed catastrophically on VSTS: test results - dotnettests_ios (no summary found). Html Report (VSDrops) Download ❌ dotnettests tests (MacCatalyst)🔥 Failed catastrophically on VSTS: test results - dotnettests_maccatalyst (no summary found). Html Report (VSDrops) Download ❌ dotnettests tests (macOS)🔥 Failed catastrophically on VSTS: test results - dotnettests_macos (no summary found). Html Report (VSDrops) Download ❌ dotnettests tests (Multiple platforms)🔥 Failed catastrophically on VSTS: test results - dotnettests_multiple (no summary found). Html Report (VSDrops) Download ❌ dotnettests tests (tvOS)🔥 Failed catastrophically on VSTS: test results - dotnettests_tvos (no summary found). Html Report (VSDrops) Download ❌ framework tests🔥 Failed catastrophically on VSTS: test results - framework (no summary found). Html Report (VSDrops) Download ❌ fsharp tests🔥 Failed catastrophically on VSTS: test results - fsharp (no summary found). Html Report (VSDrops) Download ❌ generator tests🔥 Failed catastrophically on VSTS: test results - generator (no summary found). Html Report (VSDrops) Download ❌ interdependent-binding-projects tests🔥 Failed catastrophically on VSTS: test results - interdependent-binding-projects (no summary found). Html Report (VSDrops) Download ❌ introspection tests🔥 Failed catastrophically on VSTS: test results - introspection (no summary found). Html Report (VSDrops) Download ❌ linker tests (iOS)🔥 Failed catastrophically on VSTS: test results - linker_ios (no summary found). Html Report (VSDrops) Download ❌ linker tests (MacCatalyst)🔥 Failed catastrophically on VSTS: test results - linker_maccatalyst (no summary found). Html Report (VSDrops) Download ❌ linker tests (macOS)🔥 Failed catastrophically on VSTS: test results - linker_macos (no summary found). Html Report (VSDrops) Download ❌ linker tests (tvOS)🔥 Failed catastrophically on VSTS: test results - linker_tvos (no summary found). Html Report (VSDrops) Download ❌ monotouch tests (iOS)🔥 Failed catastrophically on VSTS: test results - monotouch_ios (no summary found). Html Report (VSDrops) Download ❌ monotouch tests (MacCatalyst)🔥 Failed catastrophically on VSTS: test results - monotouch_maccatalyst (no summary found). Html Report (VSDrops) Download ❌ monotouch tests (macOS)🔥 Failed catastrophically on VSTS: test results - monotouch_macos (no summary found). Html Report (VSDrops) Download ❌ monotouch tests (tvOS)🔥 Failed catastrophically on VSTS: test results - monotouch_tvos (no summary found). Html Report (VSDrops) Download ❌ msbuild tests🔥 Failed catastrophically on VSTS: test results - msbuild (no summary found). Html Report (VSDrops) Download ❌ sharpie tests🔥 Failed catastrophically on VSTS: test results - sharpie (no summary found). Html Report (VSDrops) Download ❌ windows tests🔥 Failed catastrophically on VSTS: test results - windows (no summary found). Html Report (VSDrops) Download ❌ xcframework tests🔥 Failed catastrophically on VSTS: test results - xcframework (no summary found). Html Report (VSDrops) Download ❌ xtro tests🔥 Failed catastrophically on VSTS: test results - xtro (no summary found). Html Report (VSDrops) Download ❌ Tests on macOS Sonoma (14) tests🔥 Failed catastrophically on VSTS: test results - mac_sonoma (no summary found). Html Report (VSDrops) Download ❌ Tests on macOS Sequoia (15) tests🔥 Failed catastrophically on VSTS: test results - mac_sequoia (no summary found). Html Report (VSDrops) Download ❌ Tests on macOS Tahoe (26) tests🔥 Failed catastrophically on VSTS: test results - mac_tahoe (no summary found). Html Report (VSDrops) Download SuccessesmacOS testsLinux Build VerificationPipeline on Agent |
3 participants
This PR adds end-to-end
dotnet testintegration tests for all Apple platforms (iOS, tvOS, Mac Catalyst, macOS) using the MTP (Microsoft Testing Platform) test templates.Changes
tests/dotnet/UnitTests/DotNetTestTest.cs: New parameterized test that creates a project fromdotnet new <template>, replaces the generated tests with a single passing test, and runsdotnet test.tests/common/DotNet.cs: Addedtestverb support toDotNet.Execute.The test reads the
MLAUNCH_PATHenvironment variable to override the mlaunch binary for mobile platforms (iOS/tvOS), enabling testing with the MTP-enabled mlaunch build.Screenshot
Related
Depends on mlaunch MTP support: https://devdiv.visualstudio.com/DevDiv/_git/macios-mlaunch/pullrequest/734978