Skip to content

fix(conformance): runner_unreachable as a verdict is not a liveness failure - #400

Merged
jleni merged 1 commit into
mainfrom
fix/conformance-unreachable-is-a-verdict
Sep 26, 2026
Merged

jleni merged 1 commit into
mainfrom
fix/conformance-unreachable-is-a-verdict

Conversation

@jleni

@jleni jleni commented Sep 26, 2026

Copy link
Copy Markdown
Member

Fixes a regression I introduced in #392, found by the conformance leg on main
— run 36233876045.

What broke

#392 waited out any reply whose reason was runner_unreachable. Two
scenarios assert exactly that reason as the correct outcome:

first["state"] == "Unknown" && first["reason"] == "runner_unreachable"
  • crash_before_spawn_is_unknown_and_never_retried
  • crash_after_spawn_before_ack_is_unknown_and_runs_once

So the helper waited 60s for a verdict that was never going to change, then
failed with my own message:

[management] the runner never answered again after the injected crash cleared:
{"reason":"runner_unreachable","state":"Unknown","startedAt":…,"finishedAt":…}

Both passed before #392. I applied the helper to all four crash scenarios
because they shared a shape; only one of them shared the problem.

The two answers are distinguishable

shape meaning
verdict state, startedAt, finishedAt the execution is Unknown because the runner was unreachable
liveness {"error": …, "reason": …}, no state asked before the runner finished coming back; nothing recorded yet

#373's original failure was the second:
{"error":"the sandbox runner did not answer","reason":"runner_unreachable"}.
The ones above are the first. reason alone cannot tell them apart, and #392
keyed on reason alone.

Now keyed on the presence of state.

This is the class #399 documents

Merged the same day: the right construct — wait for the runner — pointing at
the wrong referent. reason looked like the thing that identified a liveness
failure and was not.

It is also the case for the rule in #399 about conformance: nothing in cargo test runs these scenarios, and a unit test over the helper would have used
whichever body shape I already believed in.

Verification

clippy --all-targets --all-features -D warnings clean, fmt clean. The
scenarios themselves only run on the conformance leg, so this needs
ci:conformance on the PR
— which is the lever #399 says to pull and nobody
has.

…ailure

#392 waited out any reply whose `reason` was `runner_unreachable`. Two
scenarios assert exactly that reason as the correct outcome —
`crash_before_spawn_is_unknown_and_never_retried` and
`crash_after_spawn_before_ack_is_unknown_and_runs_once` — so the helper waited
60s for a verdict that was never going to change, then failed with
"the runner never answered again after the injected crash cleared".

Both passed before #392. Caught by the conformance leg on main, which is the
only thing that runs these.

The two answers are distinguishable and #392 did not look:

- **Execution record** — carries `state`, `startedAt`, `finishedAt`. The
  execution is `Unknown` *because* the runner was unreachable. That is the
  verdict.
- **Error envelope** — `{"error": …, "reason": …}`, no `state`. The runner has
  not finished coming back from the injected crash and was asked before there
  was anything to record. That is the liveness window #373 flaked on, and the
  only thing that should be waited out.

Keyed on the presence of `state` now, which is what separates them.

This is the class documented in #399 the same day: the right construct — wait
for the runner — pointing at the wrong referent, because `reason` alone does
not say which of the two answers arrived.
@jleni jleni added the ci:conformance Run the k3s conformance gate on this PR (it is opt-in; it always runs on merge to main) label Sep 26, 2026
@jleni
jleni merged commit e6caeb5 into main Sep 26, 2026
13 of 22 checks passed
@jleni
jleni deleted the fix/conformance-unreachable-is-a-verdict branch September 26, 2026 12:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci:conformance Run the k3s conformance gate on this PR (it is opt-in; it always runs on merge to main)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant