RFC: add local adaptive-refresh replay tooling#2029
Open
hhh2210 wants to merge 6 commits into
Open
Conversation
Add local tooling for evaluating candidate refresh-timing policies against real usage traces. The harness builds on the existing adaptive refresh policy and interaction-advance behavior. New targets: - AdaptiveReplayKit (library, pure Foundation): JSONL trace schema (decision / menuOpen / refreshCompleted / timerAdvanced), strict-by-default parser with tolerant fallback, best-effort serialized writer, ReplayPolicy protocol with an advancesOnInteraction capability flag, deterministic ReplayEngine, ReplayMetrics (staleness percentiles, refresh cadence, constrained compliance, interaction-advance count), and baseline policies (mirrored adaptive table, fixed 2/5/15/30m, manual). - AdaptiveReplayCLI (executable): thin shell that parses arguments, replays a trace against one or more policies, and prints a table or JSON. App-side wiring is additive and remains off unless the adaptiveRefreshTraceEnabled defaults key is set: - AdaptiveRefreshTraceRecording writes local JSONL records containing only timestamps, signal values, reasons, delays, and event kinds. - recordDecision, recordMenuOpen, recordRefreshCompleted, and recordTimerAdvanced cover the timer tick and interaction-advance paths. - AdaptiveRefreshPolicy+ReplayAdapter lets replay tests compare the library mirror with the production policy. Tests cover hand-computed replay metrics, interaction advances, constrained policy boundaries, parser round trips, and each recorder call site. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Wires CodingActivityProbe into decision-event trace recording, gated on adaptiveRefreshTraceEnabled: samples seconds-since-last-transcript-write for Codex/Claude only while tracing is on, and records it as two optional fields that are never fed into AdaptiveRefreshPolicy. Bounds the probe's directory walk (today+yesterday for Codex, one level under Claude project dirs) after measuring ~15-23ms unbounded vs ~5-7ms bounded against this machine's real ~/.codex and ~/.claude trees. AdaptiveReplayCLI gains an informational activity-coverage summary line.
Extends CodingActivitySample with three per-CLI fields read from the same stat pass as the A-layer seconds-since-activity: session duration (mtime - creationDate), transcript byte size, and a count of transcripts modified in the last 5 minutes. Consolidates recordDecision's two flat activity params into a single activitySample: CodingActivitySample? parameter now that it carries eight fields instead of two. Adds a README documenting the harness and why token-level (C layer) probing of rate_limits/usage stays deferred until offline analysis justifies it.
hhh2210
force-pushed
the
codex/adaptive-refresh-replay-upstream
branch
from
a70eb5e to
daef5ee
Compare
Contributor
There was a problem hiding this comment.
Pull request overview
This PR introduces an opt-in, local-only adaptive-refresh trace + replay harness intended to collect measured evidence about refresh policy behavior (including segmentation of unobserved gaps, replay against baseline/candidate policies, and auditing recorded schedule decisions) without changing production defaults or enabling any network/analytics path.
Changes:
- Adds
AdaptiveReplayKit(Foundation-only) with trace parsing, replay simulation, legacy gap segmentation, metrics, activity-coverage stats, and recorded-schedule auditing. - Adds
AdaptiveReplayCLIas a thin wrapper to run replay/audit locally and emit tabular or JSON reports. - Adds an off-by-default in-app recorder (
adaptiveRefreshTraceEnabled) plus a stat-only local coding-activity probe sampled only when tracing is enabled.
Reviewed changes
Copilot reviewed 32 out of 32 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| Tests/CodexBarTests/ReplayTraceSegmentationTests.swift | Adds tests for legacy gap segmentation behavior and segmented replay rate math. |
| Tests/CodexBarTests/RecordedScheduleAuditTests.swift | Adds tests for auditing live timer-advance evaluations/advances and linkage correctness. |
| Tests/CodexBarTests/CodingActivityReplayPolicyTests.swift | Tests replay-only candidate policy that caps unconstrained active decisions at 5 minutes. |
| Tests/CodexBarTests/CodingActivityProbeTests.swift | Tests stat-only activity probe over fake home directories (Codex + Claude). |
| Tests/CodexBarTests/ClaudeOAuthCredentialsStoreCLIStorageOwnershipTests.swift | Adjusts test harness overrides wiring for Claude keychain/CLI read behavior. |
| Tests/CodexBarTests/AdaptiveReplayTraceParserTests.swift | Adds trace JSONL parsing/compatibility tests (A-layer + B-layer fields, strict vs tolerant). |
| Tests/CodexBarTests/AdaptiveReplayPolicyMirrorTests.swift | Ensures mirrored replay policy + in-app adapter match production policy decisions. |
| Tests/CodexBarTests/AdaptiveReplayEngineTests.swift | Adds hand-computed replay engine metric tests and interaction-advance behavior tests. |
| Tests/CodexBarTests/AdaptiveRefreshTraceWriterTests.swift | Tests writer byte-cap behavior and oversized-record skipping. |
| Tests/CodexBarTests/AdaptiveRefreshTraceRecordingTests.swift | Anchors in-app recorder call sites via trace-content assertions (serialized suite). |
| Tests/CodexBarTests/ActivityCoverageStatsTests.swift | Adds tests for informational activity-coverage stats computed from raw trace records. |
| Sources/CodexBar/UsageStore+AdaptiveRefresh.swift | Records adaptive decisions (and optional activity sample) when tracing is enabled. |
| Sources/CodexBar/UsageStore.swift | Records refresh completion; records timer-advance evaluations always and advances when accepted. |
| Sources/CodexBar/StatusItemController+Menu.swift | Records menu-open events for the replay harness when tracing is enabled. |
| Sources/CodexBar/CodingActivityProbe.swift | Implements bounded, stat-only activity probe for Codex/Claude transcript metadata. |
| Sources/CodexBar/AdaptiveRefreshTraceRecording.swift | Adds opt-in recorder writing JSONL trace records into Application Support with byte cap. |
| Sources/CodexBar/AdaptiveRefreshPolicy+ReplayAdapter.swift | Adapts production policy to ReplayPolicy for in-app replay tooling. |
| Sources/AdaptiveReplayKit/ReplayTraceSegmentation.swift | Implements legacy gap segmentation + segmented replay wrapper for ReplayEngine. |
| Sources/AdaptiveReplayKit/ReplayPolicy.swift | Defines replay policy interfaces and replay-friendly thermal/input types. |
| Sources/AdaptiveReplayKit/ReplayMetrics.swift | Defines replay metrics and staleness stats computations. |
| Sources/AdaptiveReplayKit/ReplayEngine.swift | Implements deterministic replay simulation including interaction-advance modeling. |
| Sources/AdaptiveReplayKit/RecordedScheduleAudit.swift | Implements auditing of recorded timer-advance events independent of replay clock. |
| Sources/AdaptiveReplayKit/README.md | Documents how tracing, parsing, replay, auditing, and activity telemetry fit together. |
| Sources/AdaptiveReplayKit/CandidatePolicies.swift | Adds replay-only candidate policy using coding-activity signal to cap delays. |
| Sources/AdaptiveReplayKit/BaselinePolicies.swift | Adds mirrored adaptive baseline plus fixed/manual baseline policies. |
| Sources/AdaptiveReplayKit/AdaptiveRefreshTraceWriter.swift | Implements JSONL writer with async serialization and hard byte limit. |
| Sources/AdaptiveReplayKit/AdaptiveRefreshTraceParser.swift | Implements strict-by-default JSONL parsing plus tolerant variant. |
| Sources/AdaptiveReplayKit/AdaptiveRefreshTrace.swift | Defines trace record schema, kinds, and record constructors. |
| Sources/AdaptiveReplayKit/ActivityCoverageStats.swift | Computes informational activity coverage fractions from raw decision records. |
| Sources/AdaptiveReplayCLI/main.swift | Adds CLI for running replay + audit and emitting table/JSON reports. |
| Package.swift | Adds AdaptiveReplayKit library and AdaptiveReplayCLI executable products/targets. |
| docs/predictive-refresh-policy.md | Updates the decision record with implementation status and local replay follow-up evidence section. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Contributor
Author
|
@clawsweeper review |
|
🦞🧹 I asked ClawSweeper to review this item again. |
Contributor
Author
|
Proof update for head 2d43edb. This also addresses Copilot's policy-name comment.
Exact-head replay output: 
@clawsweeper re-review |
Comment on lines
+34
to
+36
| for (index, line) in text.split(separator: "\n", omittingEmptySubsequences: false).enumerated() { | ||
| let trimmed = line.trimmingCharacters(in: .whitespaces) | ||
| guard !trimmed.isEmpty else { continue } |
Comment on lines
+67
to
+70
| for line in text.split(separator: "\n", omittingEmptySubsequences: false) { | ||
| let trimmed = line.trimmingCharacters(in: .whitespaces) | ||
| guard !trimmed.isEmpty, let data = trimmed.data(using: .utf8) else { continue } | ||
| if let record = try? decoder.decode(AdaptiveRefreshTraceRecord.self, from: data) { |
|
ClawSweeper status: review started. I am starting a fresh review of this pull request: RFC: add local adaptive-refresh replay tooling This is item 1/1 in the current shard. Shard 0/1. This placeholder means the worker is alive and reading the current context. I will edit this same comment with the actual review when the claws are done clicking. Crustacean status: shell secured, claws on keyboard, evidence pebbles being sorted. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Decision requested
This RFC asks whether upstream should own a local replay harness for Adaptive refresh and, if so, whether the pure replay tools should be separated from runtime trace collection and the coding-activity probe. It does not change the production policy or make Adaptive the default.
The decision record in #1739 requires measured evidence before a default change, and #1861 shipped the current opt-in policy. This branch adds tooling to collect and audit that evidence.
Refs #1739 and #1861.
What is in the branch
AdaptiveReplayKit, a Foundation-only library for parsing traces, replaying policies, segmenting unobserved gaps, calculating metrics, and auditing recorded schedule decisions.AdaptiveReplayCLI, a thin command-line wrapper around the library.Diff shape
Relative to
main, this is 32 files with 4,147 insertions and 55 deletions. It is a large review surface, so the table separates runtime wiring from the offline harness and its tests.The keychain test change wraps an existing suite in no-UI test overrides so the full validation does not read real credentials; its net change is five lines. If upstream wants a smaller sequence, the natural split is replay kit and CLI, runtime recording, then the activity probe and replay-only candidate.
Local data boundary
Tracing is disabled unless the
adaptiveRefreshTraceEnableddefaults key is set. When enabled, the app appends JSONL records in its Application Support directory. The writer has a hard 10 MiB cap and stops before a record would exceed it; it does not rotate, truncate, or upload the file.The activity probe reads file metadata only: modification time, creation time, size, and the number of recently modified transcripts. It does not read transcript contents or store file paths, project names, account information, credentials, provider responses, or menu content.
There is no network or analytics path for these records. The raw trace used below remains local and is not part of this PR. The decision record now proposes the exact file location, manual disable/delete commands, and a one-traced-process-per-file constraint. There is no retention or deletion UI; accepting that boundary is one of the decisions requested by this RFC.
Replay evidence
The frozen single-machine trace contains 1,780 records (SHA-256
b1e4aa33180b7c177293eb9ed16b45e24e026d259600fba2b1b67b931b904f0b). The legacy-gap heuristic split it into 28 observed segments and excluded 26.10 hours of unobserved wall time. This is not a causal classification: the trace cannot distinguish sleep or reboot from a long refresh or event-loop stall.Activity metadata was available for 462 of 733 recorded decisions. Of those 462 samples, 185 were inside the five-minute active window.
On this trace, current Adaptive scheduled 49.8% fewer simulated refreshes than the fixed five-minute baseline. The replay-only candidate added two refreshes, removed the four active-delay violations, and reduced p50 menu staleness by three seconds. It did not improve p95.
The fixed five-minute baseline had a p95 menu staleness of 281 seconds, compared with 1,093 seconds for both Adaptive variants. This trace therefore supports the scheduled-work reduction but does not support making Adaptive the default.
The frozen trace predates the new evaluation event. It contains 46 recorded timer advances and zero timer-advance evaluations, so it cannot validate accepted versus rejected live comparisons. A short instrumented run of the equivalent pre-rebase patch series produced two evaluation records, one accepted decision, and zero payload, decision, or menu-link mismatches. That checks the audit path; it is not additional statistical evidence.
These numbers count simulated refreshes and observed menu staleness on one machine. They do not measure battery savings, user time saved, provider API requests, or cost.
Validation
Exact candidate:
2d43edbb781fa520357c75fd6938f68d55e93bc4make checkpassed, including SwiftFormat and strict SwiftLint.make testpassed all 604 selections across 51/51 groups.swift build --product AdaptiveReplayCLIpassed.CODEXBAR_SIGNING=adhoc ./Scripts/package_app.sh debugproduced a bundle for this exact commit;codesign --verify --deep --strictpassed.Known limitations
Review questions
This PR does not propose a new production policy, persistent analytics, a default change, or use of the activity signal at runtime.