test(threads): complete retired corpus recovery evidence

[BunsDev]

Context

Engineering evidence for #886 and #888; does not close either root.
Base 5d662749b734430f84b15c73a03637e106c06140; head
fcab0eaa62955bb0d2b1240a13e323fa4104d8f7.

Acceptance: exercise two actual daemon scheduler workers on restarted,
publicly submitted migrated corpus state; refuse a canonical unsupported
identity declaration through real migration and live intake; retain exact
corpus/submission/terminal evidence. Non-goals: governance, principal-write
credentials, production scheduling changes, global historical audit repair,
Threads pin/workflow changes, and Automations leadership work in #1042.

Consulted Coven AGENTS/CONTRIBUTING, api.rs, ward_migrate.rs, existing
scheduler/terminal/corpus tests; Threads AGENTS, manifest, Phase-5 decision,
typed approval/audit contracts, synthetic corpus, E2E contract and compatibility
manifest.

Implementation

• An optional authenticated workers: 2 test-clock tick starts two concurrent
  calls to the production Threads scheduler. The outer clock-control lock,
  scheduler pass lock, decision claims and final-commit checks remain in place.
  Other counts fail closed; default single-worker responses are unchanged.
• A feature/active-fixture-only checkpoint proves both workers entered before
  the existing final-commit pause is released. Two ordinary HTTP ticks would
  serialize at the clock-control lock and would not prove this contention.
• The repository-generated familiar-review corpus is migrated and submitted
  normally, restarted pending, then contended: one apply intent, one typed
  approval, per-worker completions [0, 1], exact target bytes and stable audit
  history after another restart.
• The corpus's unknown-identity-fact declaration is refused by actual
  ward migrate --apply, leaves its original file and absence of backup intact,
  and cannot stage through live intake.
• Successful corpus scenarios export the exact synthetic case/pending record
  and source hashes, plus the original daemon audit rows. Existing corpus veto,
  downtime divergence, unavailable identity and supersession cases are mapped,
  not replaced with generic cases.
• Recovery documentation explicitly distinguishes the existing pending-item
  classifier from a missing global historical-opening census. Quarantine is
  not a typed close or historical repair.

Only six files changed: crates/coven-cli/src/api.rs,
crates/coven-cli/tests/threads_e2e.rs,
crates/coven-cli/tests/support/threads_corpus_closure_cases.rs,
docs/design/threads-test-clock.md,
docs/design/threads-terminal-recovery.md, docs/reference/threads-e2e.md.

Verification

• Eight selected feature E2E functions passed (15 scenario runs, including the
  eight-case corpus replay loop); four real-daemon legacy terminal cases passed.
• 159 selected binary tests passed, including bounded/authenticated worker
  controls, scheduler, typed terminal and recovery cases. The broad terminal
  selector also includes unrelated pre-existing terminal tests; this is not a
  count of 159 distinct Threads journeys.
• Strict cargo +1.95.0 clippy --locked --workspace --all-targets --features threads-test-clock -- -D warnings, formatting, diff, secret and staged
  privacy guards passed.
• Read-only review of the scoped diff found no significant issues.
• Full default workspace execution on the composed engineering candidate and
  current-clean-Threads overlay observation are being recorded separately;
  this draft does not claim their results prematurely.

The exact original corpus red was also reproduced at the preserved test-only
commit d15d493914ed85f304ef8d89bcfef528b2df781a, whose parent-to-commit diff
changes only threads_e2e.rs. Its real migration/intake returned 200 applied
instead of canonical pending scheduling, immediately writing TOOLS.md (26
bytes) and HEARTBEAT.md (22 bytes). Original Git core c3bd46bc and raw
manifest/audit/response evidence are retained. Production repair lineage
d3abad07 and the later merged #1022/#972/#1029 integration leads to the current
green corpus journey; this is not an artificially broken current daemon.

Initial test-authoring failures are retained: an expected 409 was actually the
existing typed 500 ward_config_invalid, and two separate tick requests
serialized at the fixture lock. Neither is misrepresented as a production red.

Risk and Rollback

R4 boundary evidence; no public Rust exhaustive type, credential, audit schema,
second audit store, or migration is introduced. New control behavior is
feature-gated and requires the existing private synthetic fixture and owner-local
transport. Default production scheduling is unchanged.

Rollback is this scoped commit; do not revert the landed identity/protected/
terminal authority repairs. All fixtures are repository-owned and synthetic.

Agent Handoff

The supported seven-case corpus and nine-case terminal evidence remains
distinct from unprovable interrupted apply or arbitrary historical corruption.
The latter requires an explicit inventory/operator resolution procedure, not
fabricated approval/rejection rows. Root #886 therefore retains that concrete
historical obligation.

Native CI for this new head, integrated current-core results, required stable
Threads compatibility activation, and parent review remain separate. No human
gate or root issue is closed by this PR. Same-daemon contention is not evidence
for a second production daemon or an OS-level write counter.

Signed-off-by: Val Alexander 68980965+BunsDev@users.noreply.github.com
Co-authored-by: Copilot 223556219+Copilot@users.noreply.github.com

[BunsDev]

Complete finite corpus engineering packet, for parent review

#1046 is at fcab0eaa62955bb0d2b1240a13e323fa4104d8f7.
This fills real missing coverage without inventing protected-write authority
or relabelling quarantine as a close.

Required corpus scenario	Exact source/evidence
Valid migration -> public pending -> visibility -> restart -> apply once	tests/threads_e2e.rs:394; real ward migrate --apply, public intake and canonical submission/pending binding.
Corpus veto	Replay loop at L661, vetoed: one proposal_vetoed/vetoed, null replay-match, no write.
Downtime materialized-byte drift	Same actual migrated corpus, surface-diverged: one evidence_diverged/false, drift bytes preserved.
Unavailable predicate after restart	Same corpus, identity-unavailable: one revalidation_failed/false, no candidate write.
Explicit supersession	L783: both proposals use corpus; old superseded/null, only replacement applied/true; both terminal rows exported.
Duplicate recovery workers race	New tests/support/threads_corpus_closure_cases.rs:4: two actual daemon scheduler workers enter before final-commit release; results [0,1], one intent and applied close, no new history after another restart.
Unsupported retired declaration	Supplement L108: actual canonical unknown-identity-fact, real migration refusal, unchanged source/no backup, live typed invalid-Ward refusal, no stage/opening/write authority.

All paths above are under crates/coven-cli/ at the immutable contribution
head. Separate HTTP tick requests serialize before the scheduler, so the new
bounded workers:2 feature control deliberately retains the clock lock and
starts two real processor calls. It does not weaken production serialization
or claim a syscall write counter or second-daemon leadership proof.

Original corpus-specific red is now reproduced

At historical test-only commit d15d493914ed85f304ef8d89bcfef528b2df781a,
the parent-to-commit diff changes only tests/threads_e2e.rs. Production was
unchanged, not intentionally broken for the test.

cargo +1.95.0 test --locked -p coven-cli --features threads-test-clock \
  --test threads_e2e \
  retired_corpus_scheduled_intake_survives_restart_and_applies_once -- --nocapture

Actual red: one failed / zero passed / eight filtered. Public
POST /api/v1/familiars/sage/edits returned 200 / applied, not 202/pending.
Its response and original ward_audit contain immediate TOOLS.md (26-byte)
and HEARTBEAT.md (22-byte) writes, no proposal IDs. After hashes:
fe1bc6b8e586949b9edadb2bf7d75659051369cbb9046ff1d0b8aa5994576e52
and e0f050746d220b1a31d27faa65950c6ce22911f7c64d711e3200e65c913ab808.
Historical metadata resolves core c3bd46bcadb6396db8436c47411a4d0eac17192b.
Production repair begins at d3abad07 and landed through
#1022 / #972 / #1029.
The same-named current journey and full seven-scenario matrix pass.

New-source native and current-core proof

• CI 34764378810, attempt 2: success, Windows job 103749105088,
  actual merge execution 5a3c70e8dea50febb21f93b4997d31039fd7935a.
  Feature E2E 81 passed, including both new corpus cases. All 93 unique
  complete native manifests/JUnit pass with Git core 0021fd26, no local
  override. Artifact 10321113850 SHA256
  26ad267ad72bcde60db1078e21ff9f2a26e0204f59af4cd60f43b84b588e6bd9.
• Final local composition 2f61260282ff47cb58ed0dc6aaf0d758e3110d6c includes
  the identity and protected-route contributions. Clean Threads
  56af664ff96239ecc78b37e70d9bffff1255e77d is proven as the actual Cargo
  dependency through the existing guarded runner: 103 E2E tests,
  111 complete manifests/JUnit, eight identity packets, plus 44 identity /
  38 protected / 30 terminal supplemental tests. Full locked workspace:
  4051 passed, five existing ignored, zero failures/filters, one invocation
  with parent runner --test-threads=1. Strict feature Clippy and guards pass.
• Both final local and native corpus/history packets have 13 exported
  openings and 13 single normative typed terminals, including both replaced
  and replacement IDs. No generic non-corpus substitute is used.
• Original red plus final source bundle, actual dependency metadata, full
  manifests/audit exports and logs are retained in the parent session's
  phase5-final-current-core-evidence.tar.gz, SHA256
  7b602fab74277220f828be999ba4497596d73e56944de8a9a0cd283831e5c4af.

The first native attempt remains a real preserved failure: existing same-home
CLI startup took 6718ms at store end against its unchanged five-second budget,
before fixture admission. Artifact 10320057204 SHA256
cef69ca9a750b953350f26a4ac0719465a36e1ced6e312a92288ce6d23fc61b7.
#1047 tracks this measured remaining native limitation.
The unchanged-head retry is not a production repair or a reliability claim.

Recommendation: the finite seven-scenario and original-red engineering
packet is now reviewable. Do not close this root on an agent's assertion;
parent owns acceptance and normal protected PR disposition. The separate
universal historical census/resolution obligation in #886 and
Threads required-pin/human gates remain open. No schema, principal profile,
deadline, durability or protected-rule change was made. Reverting evidence
controls must never restore the historical scheduling bypass.

[Copilot]

🟡 Changes recommended

Three moderate findings and one documentation nit remain unresolved.

Get a fresh assessment by requesting another Copilot review.

Pull request overview

Adds feature-gated Threads E2E evidence for retired-corpus recovery, concurrent scheduler workers, and unsupported identity rejection without changing default production scheduling.

Changes:

• Adds bounded two-worker test-clock scheduling and contention checkpoints.
• Expands migration, recovery, terminal, audit, and artifact assertions.
• Documents test-clock behavior and recovery classification boundaries.

File summaries

File	Summary	Final comments
docs/reference/threads-e2e.md	Documents corpus evidence outputs.	Nit (2 votes): Document worker completion order as order-agnostic; it may be [0, 1] or [1, 0].
docs/design/threads-test-clock.md	Documents multi-worker fixture behavior.	None.
docs/design/threads-terminal-recovery.md	Defines recovery classification boundaries.	None.
crates/coven-cli/tests/threads_e2e.rs	Adds migration and evidence capture helpers.	None.
crates/coven-cli/tests/support/threads_corpus_closure_cases.rs	Adds contention and unsupported-corpus journeys.	Moderate (1 vote): Improve the 5-second guard and include elapsed time in failures. Moderate (1 vote): Count all terminal proposal event types. Moderate (1 vote): Verify apply_audit rows prove exactly-once writes.
crates/coven-cli/src/api.rs	Adds bounded concurrent scheduler workers.	None.

Review details

Suppressed comments (3)

crates/coven-cli/tests/support/threads_corpus_closure_cases.rs:43

• This new 5-second wall-clock guard is the only bound for proving the second worker entered, so a busy shared runner can turn scheduler jitter into a false failure; the timeout message also omits the observed wait. Use a more generous hang guard and include the elapsed duration so failures remain diagnosable.

                    let deadline = Instant::now() + Duration::from_secs(5);
                    while scheduler_entries(fixture)? < before + 2 {
                        anyhow::ensure!(
                            Instant::now() < deadline,
                            "second daemon recovery worker did not reach the held pass lock"

crates/coven-cli/tests/support/threads_corpus_closure_cases.rs:166

• This invariant check omits proposal_rejected and proposal_vetoed. If live intake ever emitted a terminal refusal while leaving no pending proposal, the count would still be zero and this journey could falsely certify that the unsupported declaration produced no audit authority. Include all terminal proposal event types (or count every proposal audit row) in this check.

            let authority: i64 = fixture.store()?.query_row(
                "SELECT COUNT(*) FROM ward_audit WHERE event_type IN
                 ('proposal_submitted','proposal_window_opened','proposal_approved','apply_audit')",

crates/coven-cli/tests/support/threads_corpus_closure_cases.rs:83

• The final bytes and a single apply intent do not prove exactly-once writing: append_apply_audit_records_at records one apply_audit row per written surface, so a duplicate same-content apply could leave the files unchanged while adding extra write evidence. Count the apply_audit rows for this proposal and require one per corpus surface before claiming duplicate apply was prevented.

            let intents: i64 = fixture.store()?.query_row(
                "SELECT COUNT(*) FROM ward_audit WHERE proposal_id=?1 AND decision='proposal-apply-intent'",
                [id],
                |row| row.get(0),
            )?;
            anyhow::ensure!(intents == 1, "recovery created {intents} apply intents");

• Files reviewed: 6/6 changed files
• Comments generated: 1
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[BunsDev]

Normal landing preparation: forward-merged accepted main d00cc86 into head805ade0c9e478f1b28d6cbebf7147144fe0268b8. No conflict and no contribution change: the binary diff versus d00 is byte-identical to originalfcab0eaa versus5d662749 (SHA256af96faf01cc3e9ac860770dadde5985188167a23d1116e713fd8dcfd6c43ed04). Local20 automation-conformance,103 E2E and30 terminal tests, feature-complete all-target Clippy and formatting pass. Fresh CI34777893160 is running/queued, not yet acceptance. Original receipts remain intact. PR landing awaits the parent independent-review disposition and explicit merge authority; no root or human gate changed.