Skip to content

fix(cloud-agent): order transcripts turn by turn - #10673

Merged
eshurakov merged 1 commit into
mainfrom
kilo/linear-panda-i3i
Nov 12, 2026
Merged

eshurakov merged 1 commit into
mainfrom
kilo/linear-panda-i3i

Conversation

@eshurakov

@eshurakov eshurakov commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Order the Cloud Agent transcript turn by turn, so a failed prompt stays above a later successful retry, and a queued or failed follow-up stays below the turn that was running when it was sent.

Before this change, the order had several layers: store id order, then time.created with an "unconfirmed rows last" tier, then a causal repair. A failed queued prompt never gets a native row, so it kept the synthetic marker and sorted below every later row, including the retry and its answer. Replay also replaced its queue time with the device time.

Now one rule applies (orderMessageIds in packages/cloud-agent-sdk/src/storage/helpers.ts):

  • A turn is a user row plus the assistant rows that answer it (parentID, followed through assistant parents).
  • Turns sort by the server time of their user row. Rows in a turn follow the user row in their own time order.
  • A client row (synthetic) is placed by its server queue time. cloud.message.queued records it as a new queuedAt Kilo extension, live and on replay, from the persisted event timestamp.
  • Only a client row without any server time (an optimistic send before queue acknowledgement) sorts after every placed turn.
  • An assistant row whose prompt is not in the store opens its own turn, ordered by its own time.

synthetic keeps its meaning ("client row, not yet echoed by the server"). Retry, delivered-send detection, and unconfirmed-row handling still read it unchanged. The shared-session page and the admin trace viewer use the same order.

Both producers set parentID to the prompt that a step answers: Kilo v1 natively, and the v2 wrapper feed from the native message order. So a prompt that the runtime receives between steps owns the later steps, which matches what the runtime did.

Verification

  • SDK: 45 suites, 2,322 of 2,323 tests pass. The one failure is a timing test in native-sync-snapshot.test.ts. It also fails with this change stashed, while the local dev stack loads the machine.
  • New coverage: turn-order unit tests, and transport tests for a pending and a failed follow-up between two steps of a running turn, live and on replay. The failed-follow-up cases fail on the previous commit.
  • Web: src/lib/cloud-agent-next and src/components/cloud-agent-next, 147 suites, 3,081 tests.
  • Mobile: src/components/agents, 307 files, 5,684 tests.
  • SDK and web typechecks, oxlint, oxfmt, and git diff --check pass.

Local stack check (real Worker, sandbox, and Kilo v1 runtime, fake LLM)

  1. Failed prompt, then retry. I stopped the usage meter, so the first prompt failed while queued (billing_unavailable). Then I restarted the meter and sent a retry. After replay, main shows the failed prompt below the retry and its answer, with a device-clock time. This branch shows it first, at its queue time.
  2. Multi-step turn with a follow-up. Prompt: a Write tool step, then a step parked on a fake-LLM gate. I sent a follow-up during the parked step, then released the gate. The rendered order matched the native data: prompt, step 1, step 2 (both with the prompt as parent), follow-up, answer (parent = follow-up).
  3. Live prompt on an open page appeared at the end with its answer.

Not checked on the stack: a follow-up that fails while an earlier turn still streams. This needs delivery to fail during a live turn, which I could not cause locally without changing service code. The transport tests cover it. Kilo v2 was not run on the stack. Its parentID derivation was reviewed in code.

Local setup needed two workarounds that are not in this PR: trusting the sandbox egress proxy CA in the sandbox image build, and retrying a flaky local R2 publish. No production session was accessed.

Browser evidence

Failed prompt and retry after replay, on main:

main: the failed prompt sorts below the retry and its answer

The same session on this branch:

this branch: the failed prompt stays above the preparation failure and the retry

Multi-step turn with a follow-up sent during step 2:

follow-up stays below both steps of the earlier turn

@kilo-code-bot

kilo-code-bot Bot commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

Code Review Summary

Status: No Issues Found | Recommendation: Merge

Executive Summary

I reviewed the full PR at 25d61c4887 (turn-based transcript ordering: orderMessageIds, queuedAt on queue acknowledgement, and the shared/admin view adoption) and found no defects in the changed code: turns group by parentID with cycle/cross-session/missing-parent fallbacks that never drop or hide a row, an unplaced client row still sorts after every placed turn, and the queue timestamp comes from the persisted DO event (emitSynthetic(..., message.createdAt)) on live, snapshot, and replay paths.

Assumptions: [INFERENCE] Incremental git diff was unavailable (git is denied in this sandbox), so I reviewed the current full PR diff instead; all previously posted inline comments are line: null (outdated after the force push) and none describe a defect still present. Tests/typecheck were not executed in this read-only review, and the Worker/sandbox E2E gap named in the PR description (a follow-up that fails while an earlier turn still streams) remains covered only by transport tests.

Files Reviewed (12 files)
  • apps/web/src/lib/cloud-agent-next/message-ordering.ts
  • apps/web/src/lib/cloud-agent-next/message-ordering.test.ts
  • apps/web/src/routers/admin-router.ts
  • packages/app-shared/src/opencode.gen.ts
  • packages/cloud-agent-sdk/src/chat-processor.ts
  • packages/cloud-agent-sdk/src/chat-processor.test.ts
  • packages/cloud-agent-sdk/src/normalizer.ts
  • packages/cloud-agent-sdk/src/normalizer.test.ts
  • packages/cloud-agent-sdk/src/session.ts
  • packages/cloud-agent-sdk/src/session-transport.test.ts
  • packages/cloud-agent-sdk/src/storage/helpers.ts
  • packages/cloud-agent-sdk/src/storage/helpers.test.ts
Previous Review Summaries (3 snapshots, latest commit 0a43686)

Current summary above is authoritative. Previous snapshots are kept for context only.

Previous review (commit 0a43686)

Status: No Issues Found | Recommendation: Merge

Executive Summary

The incremental commit replaces the layered synthetic/confirmation ordering with turn-based ordering (orderMessageIds), and I found no new defects in it: a pending or failed queued follow-up now keeps its turn after every step of the earlier turn because turns sort by their prompt's server time, and the queued time is carried as queuedAt instead of confirming the row.

Assumptions: [INFERENCE] The turn model relies on native assistant rows carrying parentID back to their prompt (kilo2-source-messages.ts assigns the current user message per assistant step); sessions whose prompt row is not in the store (trimmed or older page) fall back to per-row turns, which the new tests document as intended. Tests, typecheck, and browser verification were not re-run in this read-only review; the Worker/sandbox E2E gap the PR description names remains unverified.

Files Reviewed (10 files)
  • apps/web/src/lib/cloud-agent-next/message-ordering.ts - no issues
  • apps/web/src/lib/cloud-agent-next/message-ordering.test.ts - no issues
  • apps/web/src/routers/admin-router.ts - no issues
  • packages/app-shared/src/opencode.gen.ts - generated, no issues
  • packages/cloud-agent-sdk/src/chat-processor.ts - no issues
  • packages/cloud-agent-sdk/src/chat-processor.test.ts - no issues
  • packages/cloud-agent-sdk/src/session.ts - no issues
  • packages/cloud-agent-sdk/src/session-transport.test.ts - no issues
  • packages/cloud-agent-sdk/src/storage/helpers.ts - no issues
  • packages/cloud-agent-sdk/src/storage/helpers.test.ts - no issues

Previous review (commit 3f68921)

Status: 1 Issue Found | Recommendation: Address before merge

Executive Summary

The queue-acknowledgement fix keeps pending rows unconfirmed, but a failed follow-up is still confirmed at its submission time, so it can render inside the output of the turn that was already running when it was queued.

Assumptions: [INFERENCE] The reachability is traced from server code (queueMessage, dueBackstopMessages, emitTerminal) and the SDK ordering helper; no production session was replayed and no running turn was observed failing a queued follow-up. The earlier fix for pending queued rows is verified in the new transport tests and the current code.

Overview

Severity Count
CRITICAL 0
WARNING 1
SUGGESTION 0
Issue Details (click to expand)

WARNING

File Line Issue
packages/cloud-agent-sdk/src/session.ts 311 Confirming a failed follow-up at its queue-acknowledgement time places it above steps the previous turn produced after it was queued, instead of after that turn's output
Files Reviewed (6 files)
  • packages/cloud-agent-sdk/src/chat-processor.ts - no new issues
  • packages/cloud-agent-sdk/src/chat-processor.test.ts - no issues
  • packages/cloud-agent-sdk/src/session.ts - 1 issue
  • packages/cloud-agent-sdk/src/session-transport.test.ts - no issues
  • packages/cloud-agent-sdk/src/storage/helpers.ts - no issues
  • packages/cloud-agent-sdk/src/normalizer.ts - targeted re-check, no issues

Fix these issues in Kilo Cloud

Previous review (commit c0edf6a)

Status: 1 Issue Found | Recommendation: Address before merge

Executive Summary

Queue-acknowledged rows lose the synthetic ordering marker, so a still-pending follow-up prompt can be sorted into the middle of the turn that is still streaming.

Assumptions: [INFERENCE] The interleaving depends on the running turn creating further assistant messages after a follow-up was queued (per-step message ids are visible in __fixtures__/session-replay.ts) and on the mobile transcript rendering queued rows inline (message-bubble.tsx shows the queued label); no production session was replayed.

Overview

Severity Count
CRITICAL 0
WARNING 1
SUGGESTION 0
Issue Details (click to expand)

WARNING

File Line Issue
packages/cloud-agent-sdk/src/chat-processor.ts 313 Clearing synthetic on queue acknowledgement orders pending queued rows by queue time, letting a follow-up prompt land between the still-streaming turn's assistant messages instead of below them

Fix these issues in Kilo Cloud

Files Reviewed (7 files)
  • packages/cloud-agent-sdk/src/chat-processor.ts - 1 issue
  • packages/cloud-agent-sdk/src/chat-processor.test.ts - no issues
  • packages/cloud-agent-sdk/src/normalizer.ts - no issues
  • packages/cloud-agent-sdk/src/normalizer.test.ts - no issues
  • packages/cloud-agent-sdk/src/session.ts - no issues
  • packages/cloud-agent-sdk/src/storage/helpers.ts - no issues
  • packages/cloud-agent-sdk/src/session-transport.test.ts - no issues

Reviewed by deepseek-flash-latest · Input: 152.9K · Output: 41K · Cached: 6.8M

Review guidance: REVIEW.md from base branch main

@iscekic iscekic left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

bot: Automated merge-risk stamp for this pull request, requested by @eshurakov.

The merge-risk judge did not clear this change, so the code review ran instead of a stamp.

What the change does

The PR makes a cloud.message.queued event carry the server's persisted queue time (normalizer.ts parses the envelope timestamp into createdAt, session.ts forwards it on both the live and replay paths) and uses it in chat-processor.synthesizeQueuedUserMessage: a new placeholder is stamped with that time and a pre-existing optimistic row has its time.created replaced and its synthetic ordering marker cleared. The intent is that a failed prompt, which never receives a native message.updated, is ordered before a later retry both live and after replay; the comment in storage/helpers.ts is updated to match. Tests were added for the new-row and existing-row confirmation, for the normalizer fallback on an unreadable timestamp, and for live and replay ordering plus the DO page path.

What I checked

I ran the full @kilocode/cloud-agent-sdk Jest suite from an isolated copy of src (the checkout has no dependencies and is read-only): 45 suites, 2,309 tests, all passing, including the changed chat-processor, normalizer, storage/helpers and session-transport suites. I also ran a purpose-built ordering probe against the changed code (see the e2e evidence). The data flow from the DO is correct for this purpose: both producers of cloud.message.queued stamp it with message.createdAt (session-do.ts emitQueued, and the queued bootstrap snapshot in websocket/stream.ts), so the value is a real server queue time rather than a delivery time.

What to act on first

Clearing synthetic on every queue acknowledgement, rather than only on a terminal queued failure, is broader than the bug: it re-ranks a prompt that is still queued to its queue time. A v2/native turn produces one assistant message per step, so a follow-up queued mid-turn renders between the streaming turn's step messages (I reproduced this: u1 > a1 > q2 > a2) instead of after them (u1 > a1 > a2 > q2 before the change), then jumps down when its own native row arrives. The inline comment on chat-processor.ts asks for the marker to be kept while the message is merely queued and cleared when its terminal queued failure arrives; the same comment names the test to add (queued, not failed, prompt mid-turn with a later step message).

No security, data-loss or build finding: the parsed time is server-supplied, the parse sits at the existing normalizer boundary, and nothing destructive was added.

E2E: run partially.

The checkout has no node_modules and is read-only, so I copied packages/cloud-agent-sdk/src to a temp directory outside the checkout, vendored the two workspace packages it imports (app-shared, session-ingest-contracts), installed jest/swc/jotai/zod/react, and ran the package's own Jest config. Changed suites: chat-processor, normalizer and storage/helpers, 437 tests, all pass. Full SDK suite: 45 suites, 2,309 tests, all pass (one suite initially failed only because my copy lacked the package package.json next to src; it passes once that file is present). Targeted ordering probe: with user u1(created 900), assistant a1(1000), a queued prompt q2 taken from a cloud.message.queued envelope timestamp of 1500, and assistant a2(2000), orderMessageIds returns 'u1 > a1 > q2 > a2'; with the queued row left synthetic it returns 'u1 > a1 > a2 > q2', which confirms the mid-turn interleaving introduced by clearing the marker on queue acknowledgement. Not run: any browser flow, Worker/sandbox session, or production session. The PR itself reports full Worker/sandbox E2E as unverified, and local startup was not attempted here.

Every remark is attached to the changed line where GitHub accepts one.

This service never pushes, rebases, forces, closes, labels, or merges, and it posts nothing else on this pull request.

Comment thread packages/cloud-agent-sdk/src/session.ts Resolved
@eshurakov eshurakov changed the title fix(cloud-agent): keep failed prompts before successful retries fix(cloud-agent): order transcripts turn by turn Nov 12, 2026
Replace the layered transcript order (id order, time order with an
unconfirmed tier, causal repair, confirm-on-failure) with one rule: a turn
is a user row and the assistant rows that answer it, turns sort by the
server time of their user row, and rows in a turn follow it by time.

Queue acknowledgement records the server queue time as `queuedAt` on the
client row and keeps `synthetic`, so retry, delivery, and unconfirmed
handling keep their meaning. A queued or failed follow-up sorts after the
turn that was running when it was sent, and a failed prompt stays above a
later retry. Only a row without a server time sorts last.

The shared-session page and admin trace viewer use the same order.
@eshurakov
eshurakov force-pushed the kilo/linear-panda-i3i branch from 0a43686 to 25d61c4 Compare November 12, 2026 18:51
@eshurakov

eshurakov commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor Author

Resolved the merge conflict with main and pushed 25d61c4887 with an explicit lease against the inspected remote head. The three PR commits are now one commit. Both test sets remain intact. A comparison confirms that every added and deleted line matches the original aggregate PR diff; no runtime behavior changed during resolution.

Post-rebase checks passed:

  • SDK: 46 suites, 2,348 tests.
  • SDK and web typechecks.
  • Formatting and scoped lint for all 12 PR files.
  • Whitespace and conflict-marker checks.

All three review threads were already addressed and resolved. The existing local-stack evidence in the description still covers the unchanged behavior. I did not restart the stack. The stated failed-follow-up and v2 stack coverage gaps remain.

Final verification: all 11 required checks pass for 25d61c488741a2fc223979e74822bd3665f66396. All non-required checks also pass or are skipped; none failed or remain pending. The automated review reports no issues. The PR is open and mergeable, with no unresolved review threads. The worktree is clean. The PR was not merged.

@eshurakov
eshurakov added this pull request to the merge queue Nov 12, 2026
Merged via the queue into main with commit 71ea24d Nov 12, 2026
85 checks passed
@eshurakov
eshurakov deleted the kilo/linear-panda-i3i branch November 12, 2026 19:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants