Repository navigation
Speed up hosted relay transfers while preserving Raft authority - #2162
Conversation
|
Agent review round 1 — architecture Reviewed the design against the quorum/revocation ruling. Rejected a direct local directory writer: concurrent identity writers still require consensus. The implementation keeps mutation commits and uses an invalidated, short-lived confirmation of clean applied state for reads. Fresh singleton provenance is immutable; an established voting configuration cannot be reduced during recovery. No remaining architecture blocker was found within this review. |
|
Agent review round 2 — correctness Review found that a compacted snapshot could exclude a newly admitted learner, causing repeated full snapshot transfers. Learner admission now refreshes the snapshot before catch-up and retains the resulting durable effects. The archived large-body cases pass, including restart and staging cuts. Additional checks cover every formation turn and refusal to recreate a lost published member. Reviewed production revision: a8005c4. |
|
Agent review round 3 — security and cost Review found two gaps: a replicated revoke could bypass public-handler watch closure, and an immediately rejected upload could release its in-flight slot before storage finished. Committed-apply notifications now close watches and held polls; upload tracking lasts until actual handler completion. Both findings have red-to-green regressions. Existing quorum refusal, replay and revocation assertions remain. Validation at a8005c4: 166 relay proofs, the separate actor proof, 248 unit tests, and fresh affected Go-package checks pass. Performance acceptance and remaining end-to-end fixture work are still in progress; these agent reviews do not authorize merging. |
|
Agent follow-up review of |
|
Acceptance update: the full normal relay proof suite passes 167/167 against A live two-device chat continuation succeeded after abruptly stopping the source application, preserving history and a hash-verified workspace file. However, a larger real-conversation test has exposed intermittent 503 responses during the initial directory read, before any file transfer, including samples while the source is still running. This remains an acceptance blocker; the PR stays draft and is not ready to merge. The earlier five matched public-internet transfer pairs for |
|
Acceptance update for
Mutations remain consensus commits. The live read-interruption fix retries only marked read-certificate interruptions, restarts the whole directory read, and never replays a mutation or blob I/O. Quorum loss, expired authority, activation fencing, lost-store and revocation assertions pass unchanged. |
Test mechanisms and preserved contractsThe fixture changes do not relax quorum, revocation, replay, fencing, memory admission or erasure assertions. They remove outdated setup and transport assumptions so those properties run against the consensus implementation.
The separate Raft fixtures explicitly form the existing three-voter topology before injecting faults intended for that topology. Cold-birth capacity accounting counts actual singleton voters without lifting its memory budget; restored groups still require their actual quorum. Crash cut markers follow the same durable boundaries. Replay capture begins before the first request and preserves the original command across formation, avoiding a vacuous replay assertion. Test doubles add only the new presence/read-fence interface members. New cases verify lease expiry, activation fencing, stale signatures, off-path presence, singleton formation/recovery and committed revoke interruption through observable interfaces. Watch E2E exposed and fixed two implementation errors rather than changing their assertions: duplicate direct-path version broadcasts, and closing rotated sockets before sending their final committed version. The committed-apply notifier now owns one version notification before closure; successful direct revokes also send the existing revoked event. Generic replicated revoke closure and held-I/O interruption remain intact. The aggregate final HTTP E2E suite passed on the combined source at Final aggregate verification (2026-11-02): complete HTTP E2E exit 0 against frozen combined worker SHA256 21c0b8ace6503589454b59ecd7c440dba6fc89ec0ea521c1d2d9adc2e9b30652 and WASM 5f0bb4a5810e9f72468b91339ac2b2f716281cfc76829daa5af7c9512edd64fe. All API, race, Flight, dedup, caps, pairing, link, watch, lease/idle, rotation, newcomer and persistence restart stages passed. Archive: e2e-final-combined.log and e2e-final-combined-worker-logs. Two separately counted local Wrangler transport retries; no application-status retries. Capacity proof: all 40 live, creation span 5258 ms, earliest expiry 594877 ms remaining, extra request 503. Public performance runs do not use this retry facility. |
|
Cross-platform chat acceptance depends on Furrow #23, which fixes materialization of a non-restorable Unix socket marker. The pinned 0.3.5 binary fails the minimal public reproduction; the reviewed candidate passes, including unchanged concurrent-writer and symlink protections. The candidate engine can be used for isolated end-to-end validation without merging or releasing it. This PR's dependency pin has not changed. Any successful candidate-engine handoff must be labelled as requiring that upstream fix; shipped 0.3.5 cannot be claimed to pass it. The acceptance gate is a concrete normal-app journey over the public internet: original device unavailable, real chat history and task state restored, a verified file read and edit on the receiving device, and successful publication. A download timing alone does not satisfy the gate. Intermittent live directory refusals also remain under investigation; CodeAF stays draft until repeated actual moves are reliable. |
|
Actual application acceptance update at The same private 2,278-message chat (~5.6 MB of work files, three saved tasks) was transferred through public HTTPS between two computers after stopping the source application and its background processes. Original user data was left unchanged. Both sides used the same clean native client and the explicitly recorded candidate for Furrow #23; this is not a result for the currently pinned engine alone.
These are one cold-receiver run per side, not medians or a repeatable speedup claim. Receivers had fresh disposable profiles; no chat payload or engine store was copied between computers outside the public relay. All 17 work files and the full transcript matched SHA-256; saved task contents matched after excluding derived local runtime paths. Git internals and those local paths were rebuilt during arrival. Fetch is nested in staging/preflight, so phase measurements must not all be added as disjoint stages. Cold movement made no new model calls. A separate edited-chat return succeeded on the clean candidate with the previous application stopped: 5.925 s application pipeline, all 44 workspace files, updated transcript and task state matched exactly, including the new file created on the other computer. The actual receiving agent read that file through its read tool and returned the correct contents. All five calls for this verification requested and actually served OpenRouter Repeated-move gate remains open. The next return completed download, acquisition and restoration but failed during local placement after 9.892 s. The production bucket calculation produced a 259-byte filename component ( The five-pair file-transfer measurements for c498 remain labeled c498. They will not be relabeled as bcc results. Final repeated measurements, first-model/tool latency, publication, error counts and architecture notes will be updated after the final client/relay candidate is validated. Conversation-first loading/prefetch is not implemented or measured yet. |
|
Actual repeated-move gate passes at client Six actual TUI moves alternated receiving computers, with the previous application and its background processes stopped before each confirmation. All used the same published head, without intervening model turns or file edits. After EVERY move, all 44 files, the complete transcript, and task state matched exactly. No continuation failed or required a retry.
Returning-device confirmation→visible: median 7.813 s, range 5.168–10.272 s, n=6. Separate from the single cold comparison. Some phases overlap; do not sum every column. Two driver setup faults before continuation began were preserved separately; earlier failed application attempts remain documented. After move six, the actual agent read the new verification file in the relocated workspace and returned its correct contents. First verified read-tool result: 3.964 s after prompt submission. Final reply: 7.929 s after submission. All four calls requested and actually served OpenRouter Hot fetch took 3.6–10.7 ms. The remaining cost includes directory authority and preserving apparent local changes. The omitted socket causes a false local-work plan; Furrow #23 now includes Final relay validation: 176/176 normal proof tests, full aggregate HTTP conformance, and current PR CI checks pass. Rotation kept its original two-second grace. The aggregate counted three known local transport retries separately and zero application-status retries; the earlier overloaded grace-window failure is preserved. No shared relay deployment or merge. |
Storage follow-up: balanced actual-chat timingA warm-cache comparison used the same published conversation snapshot, the same native client revision
All 12 moves completed exactly once, preserved workspace content outside derived Git internals, full conversation history and saved-task domain state, retained the externally pinned published head, and made zero model calls. Local runtime paths and Git internals are rebuilt, so this comparison does not claim byte identity for those derived files. Verdict: no demonstrated end-to-end speed improvement from the storage follow-up. Ranges overlap, and both arms still preserve a regular-file difference at The follow-up is justified by its public socket correctness regression and by removing a false same-head socket restore plan. It must not be advertised as a measured whole-journey speedup. Furrow CI passes on Linux, macOS and Windows. The final relay transfer comparison is running separately. Perceived-speed implication: hot snapshot fetch is already milliseconds. Selected-chat prefetch could hide cold fetch time, but it cannot remove authority checks or local-work preservation. Conversation-first selective network hydration and background file access are not implemented by these PRs. |
Final internet transfer comparison — clean candidateFull cold download and materialization: 9.578 → 3.423 seconds median (64.3% lower), five matched pairs. The ranges do not overlap.
Method and correctness. Two physical computers alternate sender direction across five baseline/candidate pairs. Each side starts a fresh disposable identity and transfers 2,000 high-entropy files totaling 52,428,800 plaintext bytes through public HTTPS. Baseline source is Each side completed 160 HTTP requests: all 2xx, zero request/body errors, with public-edge response identifiers on every completed request. No application-status retries. Another 40 requests per side were deliberately cancelled: baseline 22 non-success statuses/32 request-or-body errors; candidate 1 non-success/35 errors. These cancellation outcomes are excluded from completed-request success counts and are not hidden. Unchanged-head reuse makes zero payload requests; it is a cache check, not transfer throughput. Resumed downloads can redownload frames; no efficient byte-range-resume claim. Limiter and relevance. The cold payload-request span fell from median 8.329 to 1.715 seconds. Local seal and hash verification stayed similar. This supports less time awaiting relay/network operations, but does not separate WAN latency from server coordination. Independent source activity was recorded and pairs were interleaved. This is an actual full-file restoration benchmark; actual TUI chat movement and model/tool timing are recorded separately in the six-move acceptance and the balanced storage comparison. Usable transferred project. The actual copied web workspace was also served and opened in a real browser on the receiving device. All three HTML pages reached complete rendering, with no JavaScript exceptions, no external network requests and successful project-resource responses. The browser's optional favicon request returned 404; no project asset was missing. The restored JavaScript file passed its syntax check. Private screenshots are retained locally and are not published because they contain user workspace material. This functional check is separate from benchmark timing. Limits. Chat-size/team-size/task-count scaling and fleet concurrency are not established by these results. The real chat has 2,278 messages and three saved historical tasks. “Previous device unavailable” here means its complete disposable application process group was stopped, not physical power-off or a deliberately elected native-leader failure. |
|
@AbirAbbas please review this PR and Furrow #23 for merge. The review package is complete: final five-pair internet measurements, actual repeated chat movement and agent file actions, balanced storage comparison, and test-fixture/security rationale. Current CI passes on both PRs. Merge order matters: Furrow review/merge/release first; then update CodeAF's released-engine pin and rerun normal installed-engine acceptance before merging this PR. Native tests currently use the fixed Furrow candidate; the default 0.3.5 pin is unchanged. Raft mutation ordering, no-quorum refusal, revocation enforcement and concurrent-file protections remain required. The measured improvement is full restored file transfer. Actual chat movement and receiving-side read/edit/publish are validated, but instant conversation-first network hydration, arbitrary lazy file access and team-size scaling are not delivered by this PR. |
|
I brought this branch up to date with
No file you change was touched on |
#2162 replaced the native API ingress clock witness with an effects-key advance_time. Only this branch's evaluator accepts that effect; an older native computer leading the group refuses it, so worker enrollment (POST /v2/cluster/api) answered 401 against this Worker. Commit the pre-existing device_seen shape with the server's readNow() sample instead: still one synchronous clock commit before the operation, so backdated enrollment still cannot revive expired scope, and older evaluators accept it. Native ingress therefore records caller presence synchronously again; the read-path proof and the README say so. Blob authorization and watch presence stay coalesced. Verified: TestR6R15R18HostedLeaderOriginalWorkerIngressAndThinStore with dev native code against this Worker failed before and passes 3/3 now; all four native {dev, head} x Worker {dev, head} pairings pass. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Both revoke-during-upload proofs started the other device's Has before the held upload had reached its storage call, so Has sometimes ran first and the 'waiting for the actual upload' precondition failed (2 of 3, and 1 of 5, runs). The fixture now reports when the upload starts and both proofs wait for it. Assertions are unchanged; 10/10 runs pass. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
I reviewed and tested this toward merging. I found one blocking compatibility issue and one flaky proof, and pushed a separate small commit for each. Please look, and replace either one if you'd rather fix it differently. 1. The commit uses the pre-existing 2. What I checked with both commits:
Next I redeploy my head stand-in with these commits and repeat the compatibility and bench runs. On the previous head, separate base and head Workers measured Continue at p50 6.7–8.6 s against 17.7–19.2 s on base, with pair, dial and live chat unchanged. Released app/v3 and dev-20261031 clients passed pair, Continue and live chat on identities born on this relay. Then CI, the Full check and the Mac check on the new head. |
A device approval (POST /v1/dir/requests/<code>/approve) preflighted on
cached authorization. Harmless member RPCs (scheduled()) or a member
activation change invalidate that cache mid-read, which throws a read
interruption; reads recover through dispatch's fresh retry, but link
decisions are never retried, so the approval answered 503 unreachable
('cannot agree on changes right now') again and again with every device
online. On the stand-in Workers, adding a device stalled 2-10 minutes on
this branch with dev-only, mixed and head-only builds, and never on dev's
relay. Use fresh quorum authorization before admitting the decision;
activation fencing and commit-time authorization are unchanged.
The read-path proof covers first-call approval and grant delivery after a
member recovery and under one-time and continuous read interruptions, plus
revocation (401) and no quorum (503, still pending). It fails 0/3 without
this change.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Third small fix, pushed as What it fixes. The Mac check on the previous head failed at adding a 4th device, refused with "cannot agree on changes right now" for 15+ minutes while every device was online. I reproduced it on Linux against two deployed stand-in Workers, base = dev's relay and head = this branch. On dev's relay every approval went through first try. On this branch's relay approvals stalled 2–10 minutes for dev-only, mixed and head-only builds alike. So it's not a version-mix issue. The cause is the approval POST's preflight, which reused cached authorization. Harmless member RPCs ( Fresh quorum authorization on this one rare path fixes it. Activation fencing, commit-time authorization, singleton birth and formation are unchanged, and the cost is nothing measurable, since pairing is rare. Proof. The read-path proof now covers first-call approval and grant delivery after a member recovery, and under one-time and continuous read interruptions, plus revoked callers (401) and no quorum (503, still pending). It fails 0/3 with the production line reverted and passes with it. Read-path + presence 16/16 ×10, The earlier native-voting-coordinator trace also matters here: an old native coordinator admitted devices fine, so no older evaluator refuses a membership command on this path. |
|
Merging at 19153a2. Below is the evidence on this exact head, gathered while I took the PR over. Release gate (yours), done: furrow#23 merged → furrow v0.3.6 released (all six assets verified against Fixes I added (separate commits, each explained above):
CI on 19153a2: light gate, check, touched packages, Full check (3 shards) and all six cross builds are green. Linux, against two deployed stand-in Workers (base = dev's relay code, head = this branch; own storage and counters; never dev or prod):
Mac check on 19153a2 (macOS 15.6 arm64):
5-device move comparison, same Mac, base relay vs this relay (p50 of successful moves; failed attempts / total):
Every failure on both relays is the move's directory phase. Linux showed the same shape: base 31.7 / 55.1 s with 1/8 and 3/8 failed, against this branch's 31.9–44.9 s with 2–3/8 failed. At 2 devices this branch is 2–3× faster (2.8 s against 5.8–10.7 s). Known limits, filed as follow-ups: successful moves in 5-device groups are about 1.5–2.5× slower on this relay, almost all in the fetch phase. Directory-phase move failures in larger groups happen on both relays (#2327). Separately, Door: one-way as declared. Old native clients keep working on identities born here; rolling the Worker back over formed state needs compatible consensus code. The prod deploy is a separate decision. |
Non-Factory conflicts take dev: relay authorization and deferred peer formation (#2162). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Summary
Cold file download and restoration drops from 9.58 to 3.42 seconds median across five matched public-internet pairs, with every received file verified.
Consensus remains the sole directory writer. Confirmed authority is invalidated before member work and activation changes; unavailable quorum fails closed. New read fallback restarts the complete operation and pins applied state. Fix watch registration races, exact link replay/preflight, interrupted-anchor recovery and oversized copied-workspace names.
Evidence
Merge Danger
Door: one-way
Singleton birth persists immutable provenance. Older consensus code refuses that configuration; rollback needs compatible consensus code.
Blast Radius: hosted
Changes authority reuse, first-use formation, presence and committed-change notifications. Shared development and production have not been changed.
Release integration gate: Furrow #23 must be reviewed, merged and released; then update CodeAF's currently unchanged 0.3.5 pin and verify the normal installed-engine path before merging this PR for native chat acceptance. Candidate-engine testing is not proof that the default packaged dependency is fixed. Request review now; do not bypass this release gate.