feat: add offline adaptive-refresh replay tooling#2029
Conversation
a70eb5e to
daef5ee
Compare
There was a problem hiding this comment.
Pull request overview
This PR introduces an opt-in, local-only adaptive-refresh trace + replay harness intended to collect measured evidence about refresh policy behavior (including segmentation of unobserved gaps, replay against baseline/candidate policies, and auditing recorded schedule decisions) without changing production defaults or enabling any network/analytics path.
Changes:
- Adds
AdaptiveReplayKit(Foundation-only) with trace parsing, replay simulation, legacy gap segmentation, metrics, activity-coverage stats, and recorded-schedule auditing. - Adds
AdaptiveReplayCLIas a thin wrapper to run replay/audit locally and emit tabular or JSON reports. - Adds an off-by-default in-app recorder (
adaptiveRefreshTraceEnabled) plus a stat-only local coding-activity probe sampled only when tracing is enabled.
Reviewed changes
Copilot reviewed 32 out of 32 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| Tests/CodexBarTests/ReplayTraceSegmentationTests.swift | Adds tests for legacy gap segmentation behavior and segmented replay rate math. |
| Tests/CodexBarTests/RecordedScheduleAuditTests.swift | Adds tests for auditing live timer-advance evaluations/advances and linkage correctness. |
| Tests/CodexBarTests/CodingActivityReplayPolicyTests.swift | Tests replay-only candidate policy that caps unconstrained active decisions at 5 minutes. |
| Tests/CodexBarTests/CodingActivityProbeTests.swift | Tests stat-only activity probe over fake home directories (Codex + Claude). |
| Tests/CodexBarTests/ClaudeOAuthCredentialsStoreCLIStorageOwnershipTests.swift | Adjusts test harness overrides wiring for Claude keychain/CLI read behavior. |
| Tests/CodexBarTests/AdaptiveReplayTraceParserTests.swift | Adds trace JSONL parsing/compatibility tests (A-layer + B-layer fields, strict vs tolerant). |
| Tests/CodexBarTests/AdaptiveReplayPolicyMirrorTests.swift | Ensures mirrored replay policy + in-app adapter match production policy decisions. |
| Tests/CodexBarTests/AdaptiveReplayEngineTests.swift | Adds hand-computed replay engine metric tests and interaction-advance behavior tests. |
| Tests/CodexBarTests/AdaptiveRefreshTraceWriterTests.swift | Tests writer byte-cap behavior and oversized-record skipping. |
| Tests/CodexBarTests/AdaptiveRefreshTraceRecordingTests.swift | Anchors in-app recorder call sites via trace-content assertions (serialized suite). |
| Tests/CodexBarTests/ActivityCoverageStatsTests.swift | Adds tests for informational activity-coverage stats computed from raw trace records. |
| Sources/CodexBar/UsageStore+AdaptiveRefresh.swift | Records adaptive decisions (and optional activity sample) when tracing is enabled. |
| Sources/CodexBar/UsageStore.swift | Records refresh completion; records timer-advance evaluations always and advances when accepted. |
| Sources/CodexBar/StatusItemController+Menu.swift | Records menu-open events for the replay harness when tracing is enabled. |
| Sources/CodexBar/CodingActivityProbe.swift | Implements bounded, stat-only activity probe for Codex/Claude transcript metadata. |
| Sources/CodexBar/AdaptiveRefreshTraceRecording.swift | Adds opt-in recorder writing JSONL trace records into Application Support with byte cap. |
| Sources/CodexBar/AdaptiveRefreshPolicy+ReplayAdapter.swift | Adapts production policy to ReplayPolicy for in-app replay tooling. |
| Sources/AdaptiveReplayKit/ReplayTraceSegmentation.swift | Implements legacy gap segmentation + segmented replay wrapper for ReplayEngine. |
| Sources/AdaptiveReplayKit/ReplayPolicy.swift | Defines replay policy interfaces and replay-friendly thermal/input types. |
| Sources/AdaptiveReplayKit/ReplayMetrics.swift | Defines replay metrics and staleness stats computations. |
| Sources/AdaptiveReplayKit/ReplayEngine.swift | Implements deterministic replay simulation including interaction-advance modeling. |
| Sources/AdaptiveReplayKit/RecordedScheduleAudit.swift | Implements auditing of recorded timer-advance events independent of replay clock. |
| Sources/AdaptiveReplayKit/README.md | Documents how tracing, parsing, replay, auditing, and activity telemetry fit together. |
| Sources/AdaptiveReplayKit/CandidatePolicies.swift | Adds replay-only candidate policy using coding-activity signal to cap delays. |
| Sources/AdaptiveReplayKit/BaselinePolicies.swift | Adds mirrored adaptive baseline plus fixed/manual baseline policies. |
| Sources/AdaptiveReplayKit/AdaptiveRefreshTraceWriter.swift | Implements JSONL writer with async serialization and hard byte limit. |
| Sources/AdaptiveReplayKit/AdaptiveRefreshTraceParser.swift | Implements strict-by-default JSONL parsing plus tolerant variant. |
| Sources/AdaptiveReplayKit/AdaptiveRefreshTrace.swift | Defines trace record schema, kinds, and record constructors. |
| Sources/AdaptiveReplayKit/ActivityCoverageStats.swift | Computes informational activity coverage fractions from raw decision records. |
| Sources/AdaptiveReplayCLI/main.swift | Adds CLI for running replay + audit and emitting table/JSON reports. |
| Package.swift | Adds AdaptiveReplayKit library and AdaptiveReplayCLI executable products/targets. |
| docs/predictive-refresh-policy.md | Updates the decision record with implementation status and local replay follow-up evidence section. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
|
@clawsweeper review |
|
🦞🧹 I asked ClawSweeper to review this item again. Re-review progress:
|
|
Proof update for head 2d43edb. This also addresses Copilot's policy-name comment.
Exact-head replay output:
@clawsweeper re-review |
|
@clawsweeper What's your review status? Do you need to restart the review process? |
|
@clawsweeper review |
|
Codex review: needs maintainer review before merge. Reviewed July 11, 2026, 11:33 PM ET / July 12, 2026, 03:33 UTC. Summary Reproducibility: not applicable. This PR adds an offline developer analysis capability rather than fixing a reproducible current-main defect. Its behavior is demonstrated through exact-head CLI runs and focused deterministic tests. Review metrics: 3 noteworthy metrics.
Merge readiness Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch. Risk before merge
Maintainer options:
Next step before merge
Security Review detailsBest possible solution: Ship the maintainer-sponsored offline replay tool with its single canonical policy core after the remaining exact-head test gate is green or confirmed unrelated, while keeping runtime trace collection and production policy changes outside this PR. Do we have a high-confidence way to reproduce the issue? Not applicable: this PR adds an offline developer analysis capability rather than fixing a reproducible current-main defect. Its behavior is demonstrated through exact-head CLI runs and focused deterministic tests. Is this the best way to solve the issue? Yes. Sharing the production decision table while accepting only an explicit caller-supplied trace is the narrowest maintainable design because it prevents replay drift without introducing runtime telemetry; normal exact-head checks should still gate merge. AGENTS.md: found and applied where relevant. Codex review notes: model internal, reasoning high; reviewed against 237fa70c97c7. Label changesLabel justifications:
Evidence reviewedWhat I checked:
Likely related people:
What the crustacean ranks mean
Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics. How this review workflow works
Review history (9 earlier review cycles; latest 8 shown)
|
2d43edb to
d394420
Compare
d394420 to
cd96204
Compare
|
Review update for exact head
Exact-head checks passed: Trace: 1,780 records, SHA-256 Runtime proof is no longer applicable because runtime collection is no longer in the diff. This remains single-machine, author-run evidence; it makes no battery, user-time, request-count, cost, or default-policy claim. @clawsweeper re-review |
|
🦞🧹 I asked ClawSweeper to review this item again. Re-review progress:
|
|
🦞🧹 I asked ClawSweeper to review this item again. Re-review progress:
|
|
Addressed fixed-interval validation in
Validation on this head:
@clawsweeper re-review |
|
🦞🧹 I asked ClawSweeper to review this item again. Re-review progress:
|
Add local tooling for evaluating candidate refresh-timing policies against real usage traces. The harness builds on the existing adaptive refresh policy and interaction-advance behavior. New targets: - AdaptiveReplayKit (library, pure Foundation): JSONL trace schema (decision / menuOpen / refreshCompleted / timerAdvanced), strict-by-default parser with tolerant fallback, best-effort serialized writer, ReplayPolicy protocol with an advancesOnInteraction capability flag, deterministic ReplayEngine, ReplayMetrics (staleness percentiles, refresh cadence, constrained compliance, interaction-advance count), and baseline policies (mirrored adaptive table, fixed 2/5/15/30m, manual). - AdaptiveReplayCLI (executable): thin shell that parses arguments, replays a trace against one or more policies, and prints a table or JSON. App-side wiring is additive and remains off unless the adaptiveRefreshTraceEnabled defaults key is set: - AdaptiveRefreshTraceRecording writes local JSONL records containing only timestamps, signal values, reasons, delays, and event kinds. - recordDecision, recordMenuOpen, recordRefreshCompleted, and recordTimerAdvanced cover the timer tick and interaction-advance paths. - AdaptiveRefreshPolicy+ReplayAdapter lets replay tests compare the library mirror with the production policy. Tests cover hand-computed replay metrics, interaction advances, constrained policy boundaries, parser round trips, and each recorder call site. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Wires CodingActivityProbe into decision-event trace recording, gated on adaptiveRefreshTraceEnabled: samples seconds-since-last-transcript-write for Codex/Claude only while tracing is on, and records it as two optional fields that are never fed into AdaptiveRefreshPolicy. Bounds the probe's directory walk (today+yesterday for Codex, one level under Claude project dirs) after measuring ~15-23ms unbounded vs ~5-7ms bounded against this machine's real ~/.codex and ~/.claude trees. AdaptiveReplayCLI gains an informational activity-coverage summary line.
Extends CodingActivitySample with three per-CLI fields read from the same stat pass as the A-layer seconds-since-activity: session duration (mtime - creationDate), transcript byte size, and a count of transcripts modified in the last 5 minutes. Consolidates recordDecision's two flat activity params into a single activitySample: CodingActivitySample? parameter now that it carries eight fields instead of two. Adds a README documenting the harness and why token-level (C layer) probing of rate_limits/usage stays deferred until offline analysis justifies it.
41c6c0d to
b231503
Compare
|
Maintainer verification at exact head
The contributor's private trace was not copied or published. Provider/account live testing is intentionally out of scope for this offline, caller-supplied trace tool. Hosted exact-head proof: lint, changes, Linux arm64, and Linux x64 are green. The first x64 attempt had one unrelated Claude planner availability race; its clean rerun passed the build, full Linux tests, and CLI smoke. |

Summary
Add an offline
AdaptiveReplayCLIfor comparing refresh-timing policies against caller-supplied JSONL traces. Its supporting targets are package-internal and share the production adaptive decision table through the Foundation-onlyAdaptiveRefreshCoretarget.Refs #1739 and #1861.
Scope
Maintainer refinements
Validation
Maintainer validation at exact head
b231503a53020acad599672717ef80e0207544fd:make checkpassed with zero SwiftLint violations across 1,402 files.The contributor's private 1,780-record replay remains local and hash-pinned. It found 28 observed segments; current Adaptive scheduled 49.8% fewer simulated refreshes than fixed five minutes, while the activity candidate did not improve p95 staleness. This evidence does not support a default or production-policy change.
Limitations