feat(receipts): name non-gating evidence a run failed to preserve - #197
feat(receipts): name non-gating evidence a run failed to preserve#197Arifuzzamanjoy wants to merge 1 commit into
Conversation
adrian-lorenzo
left a comment
There was a problem hiding this comment.
Thanks for the contribution! self_reported: false makes this a platform check, but it does not feed engineAndChecksSucceeded, so the run can succeed while its receipt contains a failed platform check. That contradicts the runner’s invariant that every platform check must pass and gives consumers conflicting results. Please either make transcript upload failure gate the result, or represent it as a separate non-gating evidence event. Please also test the final result/receipt behavior rather than only the helper’s output.
0affe23 to
975be20
Compare
|
went with the evidence event. type: "evidence", no self_reported, so it cannot added runner/test/transcript-upload.integration.test.ts. it asserts the state one thing that follows from that. receipts only read type = "check", so the |
975be20 to
8eb815a
Compare
|
Rebased onto current upstream. ready for re-review. |
8eb815a to
c46b38d
Compare
|
Receipt now collects the evidence event, so it names the loss instead of sitting Retitled. say the word if you'd rather it read feat(receipts). |
adrian-lorenzo
left a comment
There was a problem hiding this comment.
Thanks for the contribution!
Transcript failure is currently recorded through two separate event requests: the anonymous artifact_error, followed by the named evidence. If the first succeeds and the second hits a transient failure, the receipt keeps only the anonymous error—the behavior this PR is intended to fix. I reproduced this with responses of 200 followed by 503.
Please send both events in one batch and add a regression test for this partial-failure case. The test should also exercise the real capture path rather than recreating its wiring manually, so removing the call from main() cannot leave the tests green.
With those fixes, we can approve and merge it!
1915703 to
286f6d6
Compare
A failed transcript upload emits the anonymous `artifact_error` and a named `evidence` event in one batch, so the receipt can never keep the anonymous half alone. The receipt collects the evidence beside its checks without gating the result. `main()`'s capture phase is now the exported `captureRunResult` the tests drive, and a wiring assertion fails if `main()` stops calling it, so neither the upload nor the evidence can drift out of the phase unnoticed.
286f6d6 to
405e14d
Compare
|
Both events go out in one emit(), and appendRunEvents inserts a batch in a single |
When a transcript upload fails the run still reports succeeded. It reaches the
receipt as an anonymous +1 on activity.errors, so the record can say a run hit
an error but never that the transcript is what it lost.
The runner emits a named evidence event: type "evidence", name transcript,
reason transcript_upload_failed. Not a platform check — every
self_reported: false failed check in runner/src/index.ts feeds
engineAndChecksSucceeded, and one that did not would let a run succeed while
its receipt carries a failed platform check. The receipt collects type
"evidence" alongside its checks, with truncation disclosed the same way
checks_truncated is.
artifact_error stays and keeps feeding the error count, so the existing tally is
unchanged and the runner and API can deploy in either order.
The emit is best-effort. emit() throws on a non-429 response and that reaches
main()'s outer catch, which posts a failed result. An unguarded emit would fail
a successful run over a storage blip, which is the gating this event exists to
avoid.
What it does not do is gate the run. Gating would skip the delivery phase over a
transient storage failure and throw away correct agent work to protect its own
paper trail. That trade belongs to you, so I stopped at making the loss visible.
Worth naming for two reasons: the api and security docs describe transcripts as
durably stored, so a silent loss contradicts that; and in read-only modes the
transcript is the only evidence a run produces, since nothing gets committed.
Touches runner/src/index.ts, packages/core/src/receipts.ts, and
services/api/src/sandbox/orchestrator.ts, with tests in runner,
packages/core, and services/api/test/sandbox.test.ts.