Skip to content

feat(receipts): name non-gating evidence a run failed to preserve - #197

Open
Arifuzzamanjoy wants to merge 1 commit into
theam:mainfrom
Arifuzzamanjoy:fix/transcript-evidence-check
Open

feat(receipts): name non-gating evidence a run failed to preserve#197
Arifuzzamanjoy wants to merge 1 commit into
theam:mainfrom
Arifuzzamanjoy:fix/transcript-evidence-check

Conversation

@Arifuzzamanjoy

@Arifuzzamanjoy Arifuzzamanjoy commented Aug 22, 2026

Copy link
Copy Markdown

When a transcript upload fails the run still reports succeeded. It reaches the
receipt as an anonymous +1 on activity.errors, so the record can say a run hit
an error but never that the transcript is what it lost.

The runner emits a named evidence event: type "evidence", name transcript,
reason transcript_upload_failed. Not a platform check — every
self_reported: false failed check in runner/src/index.ts feeds
engineAndChecksSucceeded, and one that did not would let a run succeed while
its receipt carries a failed platform check. The receipt collects type
"evidence" alongside its checks, with truncation disclosed the same way
checks_truncated is.

artifact_error stays and keeps feeding the error count, so the existing tally is
unchanged and the runner and API can deploy in either order.

The emit is best-effort. emit() throws on a non-429 response and that reaches
main()'s outer catch, which posts a failed result. An unguarded emit would fail
a successful run over a storage blip, which is the gating this event exists to
avoid.

What it does not do is gate the run. Gating would skip the delivery phase over a
transient storage failure and throw away correct agent work to protect its own
paper trail. That trade belongs to you, so I stopped at making the loss visible.

Worth naming for two reasons: the api and security docs describe transcripts as
durably stored, so a silent loss contradicts that; and in read-only modes the
transcript is the only evidence a run produces, since nothing gets committed.

Touches runner/src/index.ts, packages/core/src/receipts.ts, and
services/api/src/sandbox/orchestrator.ts, with tests in runner,
packages/core, and services/api/test/sandbox.test.ts.

@adrian-lorenzo adrian-lorenzo left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the contribution! self_reported: false makes this a platform check, but it does not feed engineAndChecksSucceeded, so the run can succeed while its receipt contains a failed platform check. That contradicts the runner’s invariant that every platform check must pass and gives consumers conflicting results. Please either make transcript upload failure gate the result, or represent it as a separate non-gating evidence event. Please also test the final result/receipt behavior rather than only the helper’s output.

@Arifuzzamanjoy
Arifuzzamanjoy force-pushed the fix/transcript-evidence-check branch from 0affe23 to 975be20 Compare August 24, 2026 17:12
@Arifuzzamanjoy Arifuzzamanjoy changed the title fix(runner): extract transcript evidence check as a named platform event fix(runner): use non-gating evidence event for transcript upload failure Aug 24, 2026
@Arifuzzamanjoy

Copy link
Copy Markdown
Author

went with the evidence event. type: "evidence", no self_reported, so it cannot
land in the check list and contradict the result.

added runner/test/transcript-upload.integration.test.ts. it asserts the state
uploadTranscript returns, and that filtering emitted events by type = "check",
the way the receipt query does, never picks this one up.

one thing that follows from that. receipts only read type = "check", so the
receipt itself does not change. the loss is named in run_events but not shown on
the receipt. let me know if you want it shown there.

@Arifuzzamanjoy
Arifuzzamanjoy force-pushed the fix/transcript-evidence-check branch from 975be20 to 8eb815a Compare August 27, 2026 15:55
@Arifuzzamanjoy

Copy link
Copy Markdown
Author

Rebased onto current upstream. ready for re-review.

@Arifuzzamanjoy
Arifuzzamanjoy force-pushed the fix/transcript-evidence-check branch from 8eb815a to c46b38d Compare August 28, 2026 01:33
@Arifuzzamanjoy Arifuzzamanjoy changed the title fix(runner): use non-gating evidence event for transcript upload failure fix(runner): name the transcript a failed run could not preserve Aug 28, 2026
@Arifuzzamanjoy

Copy link
Copy Markdown
Author

Receipt now collects the evidence event, so it names the loss instead of sitting
in run_events doing nothing. tested through POST /result against the stored
receipt. that reaches packages/core and services/api, flagging it rather than
slipping it past you.

Retitled. say the word if you'd rather it read feat(receipts).

@Arifuzzamanjoy Arifuzzamanjoy changed the title fix(runner): name the transcript a failed run could not preserve feat(receipts): name non-gating evidence a run failed to preserve Aug 28, 2026

@adrian-lorenzo adrian-lorenzo left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the contribution!

Transcript failure is currently recorded through two separate event requests: the anonymous artifact_error, followed by the named evidence. If the first succeeds and the second hits a transient failure, the receipt keeps only the anonymous error—the behavior this PR is intended to fix. I reproduced this with responses of 200 followed by 503.

Please send both events in one batch and add a regression test for this partial-failure case. The test should also exercise the real capture path rather than recreating its wiring manually, so removing the call from main() cannot leave the tests green.

With those fixes, we can approve and merge it!

@Arifuzzamanjoy
Arifuzzamanjoy force-pushed the fix/transcript-evidence-check branch 2 times, most recently from 1915703 to 286f6d6 Compare August 28, 2026 13:21
A failed transcript upload emits the anonymous `artifact_error` and a named
`evidence` event in one batch, so the receipt can never keep the anonymous
half alone. The receipt collects the evidence beside its checks without
gating the result. `main()`'s capture phase is now the exported
`captureRunResult` the tests drive, and a wiring assertion fails if `main()`
stops calling it, so neither the upload nor the evidence can drift out of the
phase unnoticed.
@Arifuzzamanjoy
Arifuzzamanjoy force-pushed the fix/transcript-evidence-check branch from 286f6d6 to 405e14d Compare August 28, 2026 13:51
@Arifuzzamanjoy

Copy link
Copy Markdown
Author

Both events go out in one emit(), and appendRunEvents inserts a batch in a single
transaction, so the anonymous half can't land alone. transcript-upload.integration.test.ts
covers the 200-then-503 case and asserts what the platform stored; a call-site
assertion fails if main() stops calling the phase, since main() itself isn't
exported.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants