Cap the agent's tool results and page its file reads, so one large read no longer ends the session

[DavidCockerill]

⊙ Problem

The built-in agent added every tool result to its transcript in full, and every model request replays the transcript. One large read_file of a log, the file an investigating agent most needs, pushed the next request past the model's context window. The provider rejected it, the run ended error, and every later prompt replayed the same oversized result, so the session could not be used again (agent: no cap on tool-result size — one large read kills the session (#3068)). Reproduced on 5230512f9: a 1 MB log read was stored as a 1,057,864-byte tool message, the next request was rejected, and a follow-up prompt on the same session was rejected the same way.

❓ Your call: the requirement, as harper#3068 specifies it, with two departures. The note after a cut result lives in the tool message itself, not a system message, because the Anthropic and Bedrock backends hoist inline system messages into the top-level system prompt, away from the result. The terminal reason is a distinct lastError, not a new structured field, matching how harper#3064 ends its runs. Milestoned v5.4 (the issue said v5.3), like its output-side twin harper#3064.

Closes #3068

Docs: Document agent.maxToolResultBytes and how the agent recovers when a conversation outgrows the model's context window (documentation#720)

💡 Solution

• Every tool result is capped where it is stored. invokeTool passes each observation, tool errors and unknown-tool replies included, through the toolMode: 'auto' orchestrator's serializer at agent.maxToolResultBytes (default 65,536, the orchestrator's own default). A cut result ends with …[truncated; full result is N bytes. Ask for less: …]. The serializer now cuts on a UTF-8 boundary, so the result never exceeds the cap; before, a split character could overshoot it by two bytes.
• The file tools page instead of being cut. read_file returns whole lines from startLine up to lineCount, plus nextLine and nextOffset while the file continues. Passing both back resumes without rescanning, and a line longer than a page comes back in parts. It streams, so the 5 MiB limit is gone. read_file, tail_file and grep_files measure each page JSON-escaped against half the cap, so the loop never cuts one. Paging keeps the key-material refusal from Limit the agent's config scope to the config file and refuse key material in every filesystem scope (#3098). A page of key body alone carries no armor line, so read_file and tail_file also look back 32 KiB, longer than any PEM private key, for a BEGIN line before their window.
• A context-window rejection is retried once. The OpenAI, Anthropic and Bedrock backends set contextWindowExceeded on the error when the provider rejects a request as too long. The loop then cuts every result over 2 KiB in the newest group of tool results that has one to a 2 KiB head and retries once. If nothing is left to cut, or the retry is rejected too, the run ends error with a lastError saying what to do. No tool runs again. The same step recovers a session that an oversized result made unusable before this change.
• The system prompt is sent once. It went out both as input.system and as an inline system message, about 5.7 KB extra per request; requestTurn now sends only input.system, and toModelMessages no longer inlines it.
• A plain-object throw reaches the model as its message. Harper's BigInt.prototype.toJSON throws { message }, which the model used to see as [object Object]; the loop now uses the orchestrator's errorInfo.

⚖️ Alternatives

• Cap only, nothing else. Rejected: a cut log read gives the model no way to reach the rest of the file, and it cannot rescue a session that already holds an oversized result.
• Cap when building the request, keep full results stored. Rejected: the session row is rewritten whole on every append, so stored multi-MB results multiply write cost, and the operator would see a transcript the model never saw.
• Switch the agent to generate({ toolMode: 'auto' }). Rejected for now: the orchestrator has no destructive-call approval pause or per-turn persistence.
• Line-only paging. Adopted after review: a byte cursor beside the line cursor. Without it every page rescanned the file, and a line longer than a page could not be read past its head, so a later write_file would drop its tail.

❓ Your call: the shrink rewrites the stored transcript. The cut results stay cut even if the retry still fails, and the originals survive only in the table's audit log. The alternative is to cut only the request, which then fails again on every later turn. Easy to reverse.

❓ Your call: the shrink takes the newest group of oversized results. That undoes the turn that crossed the limit and reaches a result stored before a later prompt. The cost: a session sitting at the limit loses each fresh result while older ones survive. Taking the oldest is a one-line change.

❓ Your call: the 65,536-byte default is lower than the issue's "a few hundred KB". It matches the orchestrator's default and keeps one result at about 16–20k tokens. The 1024 floor and 1 MiB ceiling stop a typo from switching the cap off.

❓ Your call: context-window detection reads provider wording (HTTP 400/413 plus a pattern, or OpenAI's context_length_exceeded code). If a provider changes its wording, recovery stops and the run ends error as before. A false positive costs one retry with the newest results cut.

❓ Your call: the system prompt now goes only in input.system. All four built-in backends honour it; a module backend that reads only messages would now lose it rather than receive it twice.

🔧 Changes

Product and architecture tour

What reaches the transcript

How big can one tool result get, and what does the model see when it is bigger?

Before	After
A tool result was stored whole: a 1 MB log read became a 1,057,864-byte tool message, replayed on every later request.	A result is cut to agent.maxToolResultBytes (65,536 by default) before it is stored, ending with a note naming its size and how to ask for less.

No observation over agent.maxToolResultBytes is appended to the transcript: tool results, tool errors and unknown-tool replies alike.
The cap applies when a result is stored, not when a request is built, because the session row is rewritten whole on every append.

• invokeTool and capObservation — every exit of a tool call goes through the cap.
• serializeToolResult — the orchestrator's serializer, now exported and exact on UTF-8 boundaries.
• maxToolResultBytes bounds — default, floor and ceiling shared by boot config and set_agent_config.

Reading a file a page at a time

How does the agent read a log far larger than its cap?

Example: paging a 1 MB log

The model calls read_file with only a path and gets the first 286 lines, about 32 KB, with nextLine: 287 and nextOffset: 32384. It passes those back as startLine and offset for the next page, so the file is not rescanned. A minified bundle's single long line comes back in parts, continued the same way.

Outcome: the whole file is reachable, every page fits the cap, and nothing is skipped.

A page's JSON-escaped text never exceeds half the cap, less the room its path and fields need, so the loop never cuts a page and its cursor survives.
Paging keeps the fs tools' key-material refusal from #3098, including for a page that falls inside a key.

• read_file handler — line cursor, byte cursor and the per-page checks.
• readPage — whole lines within the escaped budget, or part of one long line.
• tail_file and grep_files budgets — the same budget, plus truncated and skippedFiles.

When the provider says the request is too long

What happens when a conversation outgrows the model's context window anyway?

One retry after a context-window rejection

The backend flags the error, so the loop never parses provider wording; it cuts the newest large results in the stored transcript and asks once more.

sequenceDiagram
    participant Loop
    participant Backend
    participant Session
    Loop->>Backend: generate with the transcript
    Backend-->>Loop: rejected, contextWindowExceeded
    Loop->>Session: cut newest results over 2 KiB
    Loop->>Backend: generate again
    Backend-->>Loop: answer, or a second rejection
    Note over Loop: second rejection ends the run error

Loading

A context-window rejection never ends a run before the newest oversized results are cut and the request retried once.
The retry is best effort: a long prompt or history still ends the run error, with a lastError that says to shorten the prompt or start a new session.

• generateTurn — reject, cut, retry once, or end with an actionable error.
• shrinkNewestToolResults — picks the newest group by size, so the step is idempotent.
• isContextWindowRejection — provider wording and OpenAI's code, limited to HTTP 400 and 413.

Every changed file:

• agent/loop.ts: the cap, generateTurn's retry, a single system prompt, and abort rechecks after the model answers and before completed is written.
• agent/session.ts: shrinkNewestToolResults, the locked mutator the retry uses.
• agent/tools/fsTools.ts: page budget, read_file cursor, readPage, findLineStart, grep_files skippedFiles and the tail_file window.
• agent/agent.ts: the default, boot validation with a logged fallback, the cap handed to each run, and mergeConfig exported for its tests.
• agent/operations.ts: set_agent_config rejects an out-of-range value with a 400 and accepts the key.
• agent/types.ts: maxResultBytes on the tool context and maxToolResultBytes on the config.
• config-root.schema.json and utility/hdbTerms.ts: the schema entry and AGENT_MAXTOOLRESULTBYTES, so env vars and CLI flags reach it.
• resources/models/agentLoop.ts: serializeToolResult, truncateWithMarker and errorInfo exported.
• resources/models/backendHelpers.ts: the shared wording and isContextWindowExceeded.
• components/openai, components/anthropic, components/bedrock: each error class declares the flag; Bedrock classifies the SDK error's message and never surfaces it.
• DESIGN.md and agent/DESIGN.md: the index entry, the new note, and the key-material note extended to paged reads.

✅ Verification

End-to-end route: live smoke with recorded evidence. Harper booted from dist/ with a fake OpenAI-compatible backend that returns OpenAI's real 400 context_length_exceeded envelope for any request over a byte limit, and a 1 MB logs/big.log the model reads with read_file.

Run	Limit	Result
base 5230512f9	300 KB	tool message stored at 1,057,864 bytes; request 2 rejected; session error; a second prompt rejected the same way
this branch	300 KB	read_file returns a 33 KB page with nextLine/nextOffset; both prompts completed; request 1 has one system message (24.3 KB, was 30.0 KB)
this branch	45 KB	request 2 (58,024 bytes) rejected through the real OpenAIBackend; the page shrunk to 2,046 bytes; the retry (26,860 bytes) completed; the same on the second prompt
this branch, on the base run's dead session	300 KB	the stored 1,057,864-byte result shrunk to 2,046 bytes on the first rejection; the session completed and paged the log

Unit tests (all pass on the final head):

• unitTests/agent/loop.test.js: the cap on results, errors and invented tools; one system prompt; shrink-and-retry, a second rejection, nothing to cut, cancellation, approval-resumed results, a session broken before the cap; and the real OpenAI backend through Models into the loop.
• unitTests/agent/fsTools.test.js: line and byte cursors, long lines in parts, escaped page budgets at the minimum cap, tail_file and grep_files budgets, skippedFiles, and the key-material refusals for paged reads.
• unitTests/agent/session.test.js, agent.test.js and operations.test.js: the shrink selection, boot fallback and the set_agent_config 400.
• OpenAI, Anthropic and Bedrock backend tests, backendHelpers.test.js and agentLoop.test.js: which rejections are flagged, and the serializer's exact cut.

Fails on base: the new tests, applied to 5230512f9, fail at their own assertions there. The loop tests die with the unrecovered prompt is too long, the system-prompt test sees ['system', 'user'], and the paging tests hit the old 5 MiB refusal.

After merging main (4e2148d0e, which brought #3098 into the same fs functions): unitTests/agent/*.test.js plus the backend, agentLoop and backendHelpers suites pass (500), and the 45 KB live smoke above repeats its result. A --final review of the merged state found that the look-back counted an END line just past the page start, so a page starting on a key's last body line was served; 1c3a31f33 counts only armor lines that begin before the start, with a test that fails without it. The full gates below ran before the merge; CI runs them on the merged head.

Gates before the merge (this Mac; every failure also fails on 5230512f9 in the same checkout, compared file by file):

• test:unit:main, run on 5b71d2642 (the final head differs only in a test comment), with applicationSpawn.test.js excluded because it hangs (#2538): 6,703 passing, 32 failing. The failures are in globalIsolation, resolvePreload, RuntimeModuleTracker, cliOperations, processGroupReclaim, tokenAuthentication, jsLoader, OptionsWatcher, deployStagingRetention and deployActivation; base fails the same files (31 and 52 in two runs).
• test:unit:resources, run on 474871b86 (later commits touch only agent/): 4,015 passing, 3 failing, all in sourceApplyConflictRetry, the same 3 on base.
• test:integration:all, run on 474871b86: 2,270 passing, 21 failing, in isolated-application, shutdown-drain-e2e, log-rotation-write-path, rolling-restart, cert-key-reload and record-lock-concurrency. Base fails the identical 21 tests in those six files.
• oxlint --deny-warnings, prettier --check and check:design-docs pass on the changed files.

❓ Your call: post-review deltas, not re-reviewed: grep_files's skippedFiles (about ten lines, for a finding from the pre-merge final round), and the armor look-back fix in 1c3a31f33 (five lines, the post-merge reviewer's own proposed fix), plus stubbing every Models sink in one test.
_{🤖 Generated by Claude (Anthropic); posted via @DavidCockerill.}

Related PRs: #562 independent, #1770 overlaps (same backend throw sites; its retry wrapper must keep contextWindowExceeded and not retry a flagged 400), #1835 independent, #2155 independent, #2170 independent, #2436 independent, #2446 independent, #2532 overlaps (set_agent_config input schema would need maxToolResultBytes), #2649 independent, #2752 independent, #2763 independent, #2775 independent, #2901 independent, #2906 independent, #2930 independent, #2939 independent, #2986 independent, #2990 independent, #2994 independent, #3021 independent, #3030 independent, #3035 independent, #3057 independent, #3064 overlaps (same doRun, config and backend error lines; additive merge conflict), #3069 independent, #3065 independent, #3036 independent, #2981 independent, #3070 independent, #3072 independent, #3078 independent, #3079 independent, #3082 independent, #3083 independent, #3084 independent, #3086 independent, #3093 independent, #3107 independent, #3110 independent, #3113 overlaps (adjacent CONFIG_PARAMS lines in utility/hdbTerms.ts), #3098 overlaps (merged; same fs functions, conflict resolved in this branch), #3100 independent, #3103 independent, #3085 independent
Complexity: medium

_{Review-Coverage: authored=claude; ran=gemini,codex; adjudicated=domain; blocked=cursor-grok(not-installed); declined=cursor-composer,cursor-kimi,cursor-muse; rounds=4; full=3 @ 1c3a31f}

_{Review-Attention: study ~9m (decisions: do-less-alternative, shrink-durability, read-file-unbounded, context-window-by-wording, absolute-paths-in-results; raised: open major) @ 1c3a31f}

[gemini-code-assist]

Code Review

This pull request introduces a robust mechanism to prevent agent sessions from failing due to oversized tool results or context-window overflows. It adds a configurable maxToolResultBytes parameter (defaulting to 64 KiB) to cap the size of tool results appended to the transcript. The filesystem tools (read_file, grep_files, and tail_file) have been updated to dynamically page and size their outputs to fit within this budget. Additionally, the agent loop now detects context-window rejections from LLM providers (OpenAI, Anthropic, Bedrock), automatically shrinks the newest oversized tool results in the transcript, and retries the request once. Comprehensive unit tests have been added to validate these paging, capping, and recovery behaviors. No review comments were provided, so there is no feedback to address.