Skip to content

fix(ci): make the Windows supervisor failure diagnosable, and stop readiness waits failing for machine speed #85

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
BunsDev merged 5 commits into from
Sep 4, 2026
Merged

fix(ci): make the Windows supervisor failure diagnosable, and stop readiness waits failing for machine speed #85

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
f6106b5
fix(ci): stop the Windows supervisor readiness waits failing for mach…
BunsDev Sep 3, 2026
d17d3bf
fix(ci): use a scope-independent readiness literal
BunsDev Sep 3, 2026
82b013a
fix(ci): re-pin the documented supervisor test digest
BunsDev Sep 3, 2026
6d4c305
fix(conformance): name the process whose primary token SID cannot be …
BunsDev Sep 4, 2026
e737763
fix(conformance): skip unreadable session-0 processes in the quarantine
BunsDev Sep 4, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
31 changes: 28 additions & 3 deletions .github/workflows/client-v1-conformance.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4526,12 +4526,37 @@ jobs:
typeof(WTS_PROCESS_INFO_EXW));
if (information.pUserSid == IntPtr.Zero)
{
if (information.ProcessId == 0)
// The Idle process, and the protected system processes
// that live in session 0 -- Secure System, Registry,
// and their kin -- expose no primary token SID to any
// caller, however privileged. Refusing on their
// account made the drain unrunnable on hosts that
// enable virtualization-based security, which is every
// current Windows image.
//
// Skipping them does not widen what this proves. The
// supervised identity is a local account this process
// created and logged on with CreateProcessWithLogonW,
// so its processes hold a readable token and appear in
// this enumeration with a SID to compare. A process
// whose SID cannot be read AT ALL is not one of them.
//
// Anywhere else, an owner this enumeration cannot read
// is an owner it cannot rule out, so the refusal
// stands -- and names the process, because a process
// that exited between enumeration and read and a
// permanently unreadable one demand opposite fixes.
if (information.ProcessId == 0 ||
information.SessionId == 0)
{
continue;
}
throw new InvalidOperationException(
"WTS process primary token SID query was ambiguous.");
throw new InvalidOperationException(String.Format(
CultureInfo.InvariantCulture,
"WTS process primary token SID query was ambiguous "
+ "for process {0} in session {1}.",
information.ProcessId,
information.SessionId));
}
string sid = ConvertNativeSidToString(
information.pUserSid,
Expand Down
6 changes: 3 additions & 3 deletions docs/phase1-conformance.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -998,7 +998,7 @@ The later SDK validator repin must use these exact committed file bytes:

| File | Bytes | SHA-256 |
| --- | ---: | --- |
| `.github/workflows/client-v1-conformance.yml` | 457,825 | `1904746089bfa3fe079efdc686a9a12dd0f836f4bcfdc4a0df214f9e7e6c52a4` |
| `.github/workflows/client-v1-conformance.yml` | 459,800 | `047ae05690461530e609370b3d2d5c90817d5a2ff06484f13ffef9f4d5c775f2` |
| `scripts/contract-canary.mjs` | 38,191 | `4eb4d9b693187f110343a4c1efd92e59a9705e25790845bf04b05cb5bac6cbb5` |
| `scripts/executable-resolution.mjs` | 9,154 | `31e3c412ff8c835f14522f36a59e91f4a4ba82913210ae8e3b4455217503f430` |
| `scripts/owned-temp-directory.mjs` | 6,965 | `a9c55c85cf2b7d70310d278bafd2c8e7695d66f4ae38b9c3f1f12fce0b442095` |
Expand All @@ -1023,8 +1023,8 @@ The later SDK validator repin must use these exact committed file bytes:
| `scripts/unix-producer-supervisor.test.sh` | 8,083 | `c7d2d023d4c1f1ba3cdb3da1e95b30af1763602932a91592388bb117de6bd397` |
| `scripts/phase1-windows-supervisor-build.sh` | 4,646 | `713a9e0282887ade3e243b5ba175794d74cdb02c28c38dcd41491c9505812770` |
| `scripts/phase1-windows-supervisor-install.ps1` | 1,743 | `2baab275f0bb6789884cded5f6185d00bfa5348b9e7c3ad1e5575353639101d5` |
| `scripts/windows-job-supervisor.cs` | 289,604 | `20ac2be79c6ec6ebaccdf78543b68540fe2645e37e96c6715d477f19229b943f` |
| `scripts/windows-job-supervisor.test.ps1` | 168,772 | `74df69934cf26c7d6c083ce74abe840f472613b4df39d93584f5cb6391b951bf` |
| `scripts/windows-job-supervisor.cs` | 291,329 | `08c18fa81b16f922b3fac32abec3a2f6369e5f2b9f4caa19a0b48df6302bb110` |
| `scripts/windows-job-supervisor.test.ps1` | 171,179 | `55e9cf065e2dc7cc656c6aa8cc9ea53542259d3d7eee55c368c6cf0fc6356ab9` |

Before parsing or executing SDK authority, the harness queries the verified
checkout with `git rev-parse --show-object-format`, accepts only `sha1` or
Expand Down
8 changes: 4 additions & 4 deletions phase1-conformance.lock.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -116,8 +116,8 @@
},
{
"path": "scripts/windows-job-supervisor.cs",
"blob": "521a3d44e619ce9e3da385d5632e3333c0c63198",
"sha256": "20ac2be79c6ec6ebaccdf78543b68540fe2645e37e96c6715d477f19229b943f"
"blob": "547f2db1e7cacd539f49f5c879839fa80397ac1d",
"sha256": "08c18fa81b16f922b3fac32abec3a2f6369e5f2b9f4caa19a0b48df6302bb110"
},
{
"path": "scripts/contract-canary.mjs",
Expand Down Expand Up @@ -146,8 +146,8 @@
},
{
"path": ".github/workflows/client-v1-conformance.yml",
"blob": "9f31084b5fdd0b9546f3f2010f7c66cf912aeb25",
"sha256": "1904746089bfa3fe079efdc686a9a12dd0f836f4bcfdc4a0df214f9e7e6c52a4"
"blob": "28bad32e2ad0fc849ac87e9a7b9cbd3229e9640a",
"sha256": "047ae05690461530e609370b3d2d5c90817d5a2ff06484f13ffef9f4d5c775f2"
}
],
"productionDeltas": [
Expand Down
31 changes: 28 additions & 3 deletions scripts/windows-job-supervisor.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4086,12 +4086,37 @@ private static Dictionary<uint, string>
typeof(WTS_PROCESS_INFO_EXW));
if (information.pUserSid == IntPtr.Zero)
{
if (information.ProcessId == 0)
// The Idle process, and the protected system processes
// that live in session 0 -- Secure System, Registry,
// and their kin -- expose no primary token SID to any
// caller, however privileged. Refusing on their
// account made the drain unrunnable on hosts that
// enable virtualization-based security, which is every
// current Windows image.
//
// Skipping them does not widen what this proves. The
// supervised identity is a local account this process
// created and logged on with CreateProcessWithLogonW,
// so its processes hold a readable token and appear in
// this enumeration with a SID to compare. A process
// whose SID cannot be read AT ALL is not one of them.
//
// Anywhere else, an owner this enumeration cannot read
// is an owner it cannot rule out, so the refusal
// stands -- and names the process, because a process
// that exited between enumeration and read and a
// permanently unreadable one demand opposite fixes.
if (information.ProcessId == 0 ||
information.SessionId == 0)
{
continue;
}
throw new InvalidOperationException(
"WTS process primary token SID query was ambiguous.");
throw new InvalidOperationException(String.Format(
CultureInfo.InvariantCulture,
"WTS process primary token SID query was ambiguous "
+ "for process {0} in session {1}.",
information.ProcessId,
information.SessionId));
}
string sid = ConvertNativeSidToString(
information.pUserSid,
Expand Down
74 changes: 66 additions & 8 deletions scripts/windows-job-supervisor.test.ps1
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,64 @@
$ErrorActionPreference = 'Stop'
Set-StrictMode -Version Latest

# Readiness waits are 60 seconds, written out at each site.
#
# Every one of them polls for a marker some spawned process writes, and what it
# asserts is that the marker APPEARS -- never that it appears quickly. The old
# per-site values (5-30s) were tuned on a runner that could start a
# secondary-user session in a second or two. Starting one now regularly takes
# far longer on the hosted windows-2025 image, where creating the logon session
# and loading a fresh profile is most of the cost and is paid before the
# spawned script runs its first statement. Those deadlines began firing for
# machine speed rather than for anything this suite exists to catch.
#
# Deliberately a literal at each site rather than one shared helper: several of
# these waits live inside here-strings that are written out and run as their
# own child scripts, where a function defined in this file does not exist. A
# helper reads better and fails at runtime in exactly the places that matter.
#
# This bounds patience, not behaviour: a marker that never arrives still fails,
# and the job's own 20-minute timeout still bounds the run.

# Name the cause when something throws.
#
# The supervisor wraps a quarantine failure as "Terminal producer identity
# quarantine failed" with the real fault as InnerException, and as an
# AggregateException when the producer failed too. PowerShell prints only the
# outer message, so a CI failure said which stage failed and never why. This
# prints the whole chain before the exception continues to terminate the run.
function Write-ExceptionChain {
param([Parameter(Mandatory)][AllowNull()][object]$Failure)

$exception = if ($Failure -is [Management.Automation.ErrorRecord]) {
$Failure.Exception
} else {
$Failure
}
$depth = 0
while ($null -ne $exception -and $depth -lt 12) {
Write-Host "cause[$depth] $($exception.GetType().FullName): $($exception.Message)"
if ($exception -is [AggregateException]) {
$index = 0
foreach ($inner in $exception.InnerExceptions) {
Write-Host " aggregate[$index] $($inner.GetType().FullName): $($inner.Message)"
$index++
}
}
$exception = $exception.InnerException
$depth++
}
}

trap {
Write-Host '--- windows-job-supervisor.test.ps1 failure ---'
Write-ExceptionChain -Failure $_
if ($null -ne $_.ScriptStackTrace) {
Write-Host $_.ScriptStackTrace
}
break
}
Comment on lines +53 to +60

if (-not $IsWindows -or [System.Runtime.InteropServices.RuntimeInformation]::OSArchitecture -ne 'X64') {
throw 'Windows Job Object runtime tests require Windows x64.'
}
Expand Down Expand Up @@ -513,7 +571,7 @@ if (
function Assert-ProcessExited {
param([Parameter(Mandatory)][int]$ProcessId)

$deadline = [DateTime]::UtcNow.AddSeconds(10)
$deadline = [DateTime]::UtcNow.AddSeconds(60)
do {
try {
$process = [Diagnostics.Process]::GetProcessById($ProcessId)
Expand All @@ -532,7 +590,7 @@ function Assert-BoundedTextMarker {
[Parameter(Mandatory)][string]$Expected
)

$deadline = [DateTime]::UtcNow.AddSeconds(5)
$deadline = [DateTime]::UtcNow.AddSeconds(60)
do {
try {
if ([IO.File]::Exists($Path)) {
Expand Down Expand Up @@ -1875,7 +1933,7 @@ $attack = Start-Process `
-RedirectStandardError $env:OPENCOVEN_ROOT_ATTACK_STDERR `
-PassThru
try {
$deadline = [DateTime]::UtcNow.AddSeconds(10)
$deadline = [DateTime]::UtcNow.AddSeconds(60)
while (-not [IO.File]::Exists($env:OPENCOVEN_ROOT_ATTACK_COMPLETE)) {
if ($attack.HasExited) {
throw "Root process attack failed: $(
Expand Down Expand Up @@ -2428,7 +2486,7 @@ while (-not [IO.File]::Exists($env:OPENCOVEN_HANDOFF_RACE_STOP)) {
-RedirectStandardError (Join-Path $operatorPrivateRoot 'handoff-race.stderr') `
-PassThru
try {
$raceDeadline = [DateTime]::UtcNow.AddSeconds(10)
$raceDeadline = [DateTime]::UtcNow.AddSeconds(60)
while (-not [IO.File]::Exists($raceReady)) {
if ($raceProcess.HasExited) {
throw 'Artifact replacement race exited before replacing the record.'
Expand Down Expand Up @@ -3262,7 +3320,7 @@ public static class UnsupervisedLogonProcess
"-NoLogo -NoProfile -NonInteractive -File `"$lateRegistrarScript`" -UserName `"$($serviceEscapeContext.User.UserName)`"",
$serviceEscapeContext.User.RootPath
)
$lateReadyDeadline = [DateTime]::UtcNow.AddSeconds(20)
$lateReadyDeadline = [DateTime]::UtcNow.AddSeconds(60)
while (-not [IO.File]::Exists($lateRegistrarReady)) {
try {
$lateRegistrar = [Diagnostics.Process]::GetProcessById($lateRegistrarPid)
Expand Down Expand Up @@ -3783,7 +3841,7 @@ exit 23
"-NoLogo -NoProfile -NonInteractive -File `"$failureSleeperScript`"",
$failureEscapeContext.User.RootPath
)
$failureSleeperDeadline = [DateTime]::UtcNow.AddSeconds(20)
$failureSleeperDeadline = [DateTime]::UtcNow.AddSeconds(60)
while (-not [IO.File]::Exists($failureSleeperReady)) {
try {
$failureSleeper = [Diagnostics.Process]::GetProcessById(
Expand Down Expand Up @@ -4039,7 +4097,7 @@ Start-Sleep -Seconds 300
"-NoLogo -NoProfile -NonInteractive -File `"$setupPath`"",
$Context.User.RootPath
)
$readyDeadline = [DateTime]::UtcNow.AddSeconds(30)
$readyDeadline = [DateTime]::UtcNow.AddSeconds(60)
while (-not [IO.File]::Exists($readyPath)) {
try {
$setupProcess = [Diagnostics.Process]::GetProcessById($setupPid)
Expand Down Expand Up @@ -4533,7 +4591,7 @@ Add-Type -TypeDefinition ([IO.File]::ReadAllText('$($retainedHandleSource.Replac
$retainedRootScript,
@"
`$descendant = Start-Process -FilePath '$($trustedPwsh.Replace("'", "''"))' -ArgumentList @('-NoLogo','-NoProfile','-NonInteractive','-File','$($retainedHandleScript.Replace("'", "''"))') -RedirectStandardOutput '$((Join-Path $root 'retained-stdout.txt').Replace("'", "''"))' -RedirectStandardError '$((Join-Path $root 'retained-stderr.txt').Replace("'", "''"))' -PassThru
`$deadline = [DateTime]::UtcNow.AddSeconds(10)
`$deadline = [DateTime]::UtcNow.AddSeconds(60)
while (-not [IO.File]::Exists('$($retainedPidPath.Replace("'", "''"))')) {
Comment on lines 4593 to 4595
if (`$descendant.HasExited) {
throw 'Retained Job handle descendant exited before reporting readiness.'
Expand Down
Loading