Skip to content

Commit 6a1ccff

Browse files
mason: deliver bridge memory heads with cancellable step budgets
Record the host-spelled project identity before provider rendering while keeping the subc route realpathed. The OpenCode head remains synthetic user messages, not system text. Return { messages, kind } and enforce whole-pass 5s ordinary/30s bootstrap-or-rebuild budgets. Stage bridge record writes until a final atomic, cancel-checked commit; dispatch committed observers outside the budget. Native provider behavior is unchanged. Bridge marker writes to OpenCode's separate store are intentionally omitted; document native auto-compaction decline and the coverage-first, idempotent post-publication marker follow-up. Verified with copied cc8617e module, private daemon, captured provider requests, isolated lsof inventories, 456 scoped tests and cancellation/identity/marker mutation controls. Co-authored-by: Alfonso <alfonso-magic-context@users.noreply.github.com>
1 parent 245aa97 commit 6a1ccff

10 files changed

Lines changed: 1012 additions & 122 deletions

File tree

‎docs/host-runner-integration.md‎

Lines changed: 113 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -25,8 +25,9 @@ const adapter = await createBridgeAdapter({
2525
const unsubscribe = adapter.onOutcome(async (messageId, subject, outcome) => {
2626
// Observe committed ops/unavailability; context.db remains the replay owner.
2727
});
28-
const view = await adapter.run({ sessionId, messages: rawHostWindow, kind: "model" });
29-
// Send this view, not the unchecked rawHostWindow.
28+
const { messages: view, kind } = await adapter.run({ sessionId, messages: rawHostWindow, kind: "model" });
29+
// Send this view, including its synthetic user head, not the unchecked rawHostWindow.
30+
// Log kind: "ordinary", "bootstrap", or "rebuild".
3031
unsubscribe();
3132
await adapter.dispose();
3233
```
@@ -48,11 +49,113 @@ whether the selected model, system/tool fingerprint or persisted source state ha
4849
changed. Idle-cache time and token pressure can also require a rebuilt history
4950
view, using the same rules as the in-tree provider path.
5051
`run` owns `SessionMeta`, publication, normalization through `providerIncoming`,
51-
and the bootstrap/rebuild record; it returns a `Promise<MessageLike[]>` view.
52+
and the bootstrap/rebuild record; it returns `Promise<{ messages: MessageLike[],
53+
kind: "ordinary" | "bootstrap" | "rebuild" }>`.
5254
Supply the complete raw host window in append order with its real identities,
5355
not just the new tail or a previous provider projection. No private type or
5456
mutable output parameter is needed.
5557

58+
### Head placement and project identity
59+
60+
The OpenCode provider's head is **inside `messages`**, not a system contribution.
61+
Its first two entries, m[0] and m[1], are id-less synthetic **user** messages, just
62+
as in the in-tree plugin. m[0] carries project docs, the user profile,
63+
`<session-history>` and `<project-memory>`; m[1] carries incremental history and
64+
memory corrections when present. Send both entries at the head of the provider
65+
window. Do not filter them out, turn them into system text, persist them as raw
66+
host history, or add a second memory injection. `run` does not drop a separate
67+
system field: there is no such head field on the OpenCode provider answer.
68+
69+
Pass `projectRoot` **exactly as the host spells it; do not realpath it first**.
70+
The shared identity resolver uses `git:<root-commit>` for repositories with
71+
history and `dir:<MD5-prefix of path.resolve(root)>` otherwise. A trailing slash
72+
does not change the key, but symlink spellings and macOS `/var` versus
73+
`/private/var` can. Transport still binds a realpath for filesystem authority.
74+
Before provider rendering the bridge records the host-resolved identity in
75+
`session_projects` under its instance's harness. ck-mc prefers that binding to
76+
recomputing the identity from the canonical route. Host watermarks and all
77+
project-scoped module reads use that identity; docs use the equivalent bound
78+
filesystem directory. Re-keying existing directory memories is not part of this
79+
API. Seed/write memories using the same host spelling and shared resolver.
80+
81+
Setup's `answer: "ready"` inserts placeholders; the bootstrap's final status pass
82+
renders the real head even with **zero compartments** (stored history summaries)
83+
and no historian execution (background model calls that create those summaries).
84+
Memory and profile injection need `memory.enabled` (default true),
85+
`memory.injection_budget_tokens` (default 4,000) and
86+
`memory.user_profile_budget_tokens`; docs need `dreamer.inject_docs` (default true). The provider lane requires
87+
`compaction.enabled` (default true); disabling it refuses setup, rather than
88+
silently serving a memory-less provider window.
89+
90+
The TS facade's `configPath` does not configure an independently launched ck-mc.
91+
At this pin ck-mc reads its user tier from
92+
`$XDG_CONFIG_HOME/cortexkit/magic-context.jsonc` (or `.json`) plus the project's
93+
`.cortexkit` tier; it does **not** read `MAGIC_CONTEXT_CONFIG_PATH`.
94+
`MAGIC_CONTEXT_STORAGE_DIR` selects storage, not head settings. An env-only
95+
launch still gets the default enabled head, but explicit overrides must be
96+
installed in ck-mc's isolated XDG config tier as well. Staged ck-mc `cc8617ec`
97+
has identical Rust provider/identity/render/config code to `245aa977`; it does
98+
not need rebuilding for this fix.
99+
100+
### Whole-step budgets and cancellation
101+
102+
The facade classifies the pass using the provider adapter's own bootstrap and
103+
rebuild checks (source revert, model/system/tool manifest, explicit refresh and
104+
idle cache). `kind: "model"` remains the input; it is not a budget hint.
105+
106+
| Result/error `kind` | Whole-pass budget |
107+
| --- | --- |
108+
| `ordinary` | 5,000 ms |
109+
| `bootstrap` | 30,000 ms |
110+
| `rebuild` | 30,000 ms |
111+
112+
The budget starts on entry to `run` and covers preflight, all provider/ck-mc calls,
113+
assembly and the final host-record commit. Individual RPC limits can refuse
114+
earlier. At the deadline `run` rejects with `StepBudgetExceededError`, carrying
115+
`{ code: "step_budget_exceeded", budget_ms, kind }`. Errors from other pass
116+
failures also carry the classified `kind`. The caller may then use
117+
`isFallbackAdmissible`; timeout itself never authorizes unchecked raw history.
118+
119+
Bridge host-record writes are staged until a single final atomic transaction.
120+
Cancellation is checked between asynchronous stages, before each record write,
121+
and immediately before COMMIT, including a clock check when synchronous SQLite
122+
work prevents the timer from running. An expired transaction rolls back whole;
123+
no outcomes, served state, request fences or usage from an unserved timed-out
124+
pass are published. Request fences associate a fresh RPC ID with its lineage and
125+
newest ordinal, preventing stale replies from overwriting a newer view. A retry
126+
reloads the prior durable record. Transport receives
127+
the cancellation signal; a remotely finishing ck-mc reply is discarded and
128+
cannot resume host writes. Session/project binding and module state-sync are
129+
independent authority metadata, not proof that a provider step was served.
130+
131+
Bridge `onOutcome` is a fire-and-forget notification **after successful commit
132+
and publication**, outside the step budget. It emits nothing for cancelled
133+
passes, is not awaited by `run`, and has no ordering guarantee relative to
134+
`run`'s resolution. Observer failures are isolated and logged once per session.
135+
`dispose` drains outstanding notifications before closing handles.
136+
137+
### Raw host-history markers and open items
138+
139+
Budgeted bridge passes **do not invoke the native raw-history marker writer**.
140+
That writer commits OpenCode's own database independently of `context.db`; it
141+
cannot safely trim raw history inside a transaction that might roll back the
142+
bridge view at the deadline. Coverage is still committed in the bridge's durable
143+
view. Marker advancement is an optimization, never a condition for serving a
144+
step.
145+
146+
Without markers, OpenCode keeps handing the full raw history to each step. The
147+
returned compacted view stays correct, but per-step host work grows with the
148+
session. The bridge integration must also **keep OpenCode's native
149+
auto-compaction declined while Magic Context owns compaction**; the facade does
150+
not install the host's compaction-decline hook on the integration's behalf.
151+
152+
Open follow-up: add bridge marker advancement as a separate step **after view
153+
publication**, outside the step budget and idempotent on retry. Commit
154+
`context.db` coverage first, then advance the `opencode.db` marker. A crash
155+
between these writes must leave an untrimmed host (safe), never a trimmed host
156+
whose coverage Magic Context has not recorded. The native in-tree marker path
157+
is unchanged.
158+
56159
The store is an independently owned handle opened by the shared storage module,
57160
with the ordinary WAL/NORMAL policy, foreign keys, 5-second busy timeout, private
58161
permissions and write-transaction helpers. It does not reuse or close another
@@ -88,7 +191,7 @@ v2's host-history registration lifecycle.
88191
rejects new calls immediately, drains in-flight passes and stops the historian,
89192
then closes the module connection and its store handle exactly once. Methods
90193
other than repeated disposal refuse after disposal. Observers can unsubscribe;
91-
they are awaited after durable commit, as described below.
194+
the bridge dispatches them after durable commit, as described above.
92195

93196
### Refused provider handoffs
94197

@@ -187,10 +290,12 @@ outcomes are superseded by the durable replacement in `host_runner_views`; cover
187290
messages are not re-hooked. A lineage descent/revert can deliberately re-admit a
188291
changed suffix.
189292

190-
`OpenCodeProviderOptions.onOutcome(messageId, subject, outcome)` is optional and
191-
awaited for newly committed outcomes, after the pass commit and before publication
192-
or `run()` returning. It observes/mirrors the record; it never replaces persistence.
193-
An observer failure refuses that pass's publication but leaves the frozen outcomes
293+
The low-level native `OpenCodeProviderOptions.onOutcome(messageId, subject, outcome)`
294+
is optional and awaited for newly committed outcomes, after the pass commit and
295+
before publication or `run()` returning. The supported bridge instead dispatches
296+
observers after publication, without awaiting them, as described above. Both
297+
observe/mirror the record; neither replaces persistence.
298+
On the native path an observer failure refuses publication but leaves the frozen outcomes
194299
committed. Notifications are not an exactly-once delivery queue and are not re-sent
195300
on replay; a process can die after commit but before observation. Bridges must not
196301
depend on a separate mirror to reconstruct the served view. The SIGKILL fixture
Lines changed: 138 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,138 @@
1+
import { expect, test } from "bun:test";
2+
import { execFileSync } from "node:child_process";
3+
import { copyFileSync, mkdirSync, readFileSync, realpathSync, symlinkSync, writeFileSync } from "node:fs";
4+
import { join, resolve } from "node:path";
5+
import { resolveProjectIdentity } from "../../plugin/src/features/magic-context/memory/project-identity";
6+
import { computeNormalizedHash } from "../../plugin/src/features/magic-context/memory/normalize-hash";
7+
import { openBridgeDatabase } from "../../plugin/src/features/magic-context/storage-db";
8+
import { createBridgeAdapter } from "../../plugin/src/host-runner";
9+
import { setRawMessageProvider } from "../../plugin/src/hooks/magic-context/read-session-chunk";
10+
import type { MessageLike } from "../../plugin/src/hooks/magic-context/transform-operations";
11+
import { HermeticSubcStack } from "./rust-runner/hermetic-subc";
12+
13+
const enabled = process.env.MC_BRIDGE_MEMORY_E2E === "1";
14+
const memory = "BRIDGE_MEMORY_PROOF: the durable architecture uses one context database.";
15+
16+
function user(id: string): MessageLike {
17+
return {
18+
info: { id, role: "user", model: { providerID: "openai", modelID: "gpt-5.6" } },
19+
parts: [{ id: `${id}-text`, type: "text", text: id }],
20+
} as MessageLike;
21+
}
22+
23+
function rawHistory(session: string, rows: MessageLike[]) {
24+
return setRawMessageProvider(session, {
25+
readMessages: () => [],
26+
getStoredMessageCount: () => rows.length,
27+
getMessageCount: () => rows.length,
28+
hasMessageById: (id) => rows.some((row) => row.info.id === id),
29+
readMessageOrdinalPage: (after, limit) => rows.map((row, index) => ({
30+
id: row.info.id!, timeCreated: index + 1, contributesOrdinal: true, hasValidInfo: true,
31+
})).filter((row) => !after || row.timeCreated > after.timeCreated).slice(0, limit),
32+
});
33+
}
34+
35+
// MC_BRIDGE_MEMORY_E2E enables the real-module test with caller-supplied binary copies.
36+
// The harness starts a private daemon rather than connecting to the user's running daemon.
37+
for (const linked of [false, true]) {
38+
(enabled && (linked || process.platform === "darwin") ? test : test.skip)(`real bridge sends seeded project memory in synthetic user head on first and ordinary passes (${linked ? "symlink" : "/var versus /private/var"})`, async () => {
39+
const suppliedRoot = process.env.MC_BRIDGE_MEMORY_ROOT!;
40+
const ckMcBin = process.env.MC_BRIDGE_MEMORY_BIN!;
41+
const ckSubcBin = process.env.MC_BRIDGE_MEMORY_SUBC_BIN!;
42+
if (!suppliedRoot || !ckMcBin || !ckSubcBin) throw new Error("Supply MC_BRIDGE_MEMORY_ROOT, BIN and SUBC_BIN under a throwaway root");
43+
const taskRoot = realpathSync(suppliedRoot);
44+
const root = join(suppliedRoot, linked ? "symlink" : "spelling");
45+
for (const binary of [ckMcBin, ckSubcBin]) expect(realpathSync(binary).startsWith(`${taskRoot}/`)).toBe(true);
46+
const dataDir = join(root, "data");
47+
const storageDir = join(dataDir, "cortexkit", "magic-context");
48+
const project = join(root, linked ? "project-link" : "project-target");
49+
mkdirSync(join(root, "project-target"), { recursive: true });
50+
if (linked) symlinkSync(join(root, "project-target"), project);
51+
if (!linked && process.platform === "darwin") {
52+
expect(project.startsWith("/var/")).toBe(true);
53+
expect(realpathSync(project)).toBe(`/private${project}`);
54+
}
55+
mkdirSync(storageDir, { recursive: true });
56+
mkdirSync(join(root, "config"), { recursive: true });
57+
const configPath = join(root, "config", "selected.jsonc");
58+
writeFileSync(configPath, JSON.stringify({
59+
historian: { disable: true }, dreamer: { disable: true }, embedding: { provider: "off" },
60+
}));
61+
const fixture = process.env.MC_BRIDGE_MEMORY_DB;
62+
if (fixture) copyFileSync(fixture, join(storageDir, "context.db"));
63+
const db = openBridgeDatabase(join(storageDir, "context.db"), true);
64+
const identity = resolveProjectIdentity(project);
65+
const canonicalIdentity = resolveProjectIdentity(realpathSync(project));
66+
expect(identity).not.toBe(canonicalIdentity);
67+
expect(resolveProjectIdentity(`${project}/`)).toBe(identity);
68+
db.prepare("INSERT INTO memories (project_path, category, content, normalized_hash, status, first_seen_at, created_at, updated_at, last_seen_at) VALUES (?, 'ARCHITECTURE', ?, ?, 'active', 0, 0, 0, 0)").run(identity, memory, computeNormalizedHash(memory));
69+
console.log(`Bun ${Bun.version}; ck-mc ${execFileSync(ckMcBin, ["--version"], { encoding: "utf8" }).trim()}`);
70+
console.log(JSON.stringify({ project, identity, canonicalRoot: realpathSync(project), canonicalIdentity, trailingSlashIdentity: resolveProjectIdentity(`${project}/`) }));
71+
const requests: any[] = [];
72+
const server = Bun.serve({
73+
hostname: "127.0.0.1", port: 0,
74+
async fetch(request) {
75+
requests.push(await request.json());
76+
return Response.json({ choices: [{ message: { role: "assistant", content: "ok" } }] });
77+
},
78+
});
79+
let stack: HermeticSubcStack | undefined;
80+
let adapter: Awaited<ReturnType<typeof createBridgeAdapter>> | undefined;
81+
const rows = [user("first")];
82+
const session = `bridge-memory-${Date.now()}`;
83+
const unregister = rawHistory(session, rows);
84+
try {
85+
stack = await HermeticSubcStack.start({ dataDir, ckMcBin, ckSubcBin, startProducer: false, historianRunner: null,
86+
moduleEnv: { MAGIC_CONTEXT_CONFIG_PATH: configPath } });
87+
stack.writeModuleConfig({ historian: { disable: true }, dreamer: { disable: true }, embedding: { provider: "off" } });
88+
adapter = await createBridgeAdapter({ openCodeClient: { session: { hook: async () => {} } } as any,
89+
projectRoot: project, storageDir, configPath, connectionFile: stack.connectionFile });
90+
for (let pass = 0; pass < 2; pass++) {
91+
if (pass) rows.push(user("second"));
92+
const result = await adapter.run({ sessionId: session, messages: rows.slice(), kind: "model" });
93+
expect(result.kind).toBe(pass ? "ordinary" : "bootstrap");
94+
const view = result.messages;
95+
// Preserve the adapter's message order and roles, including its two leading
96+
// synthetic user messages, when sending the fake provider's request.
97+
const response = await fetch(`http://127.0.0.1:${server.port}/v1/chat/completions`, {
98+
method: "POST", headers: { "content-type": "application/json" },
99+
body: JSON.stringify({ model: "fake", messages: view.map((message) => ({
100+
role: message.info.role,
101+
content: message.parts.flatMap((part) => {
102+
if (!part || typeof part !== "object") return [];
103+
const block = part as { type?: unknown; text?: unknown };
104+
return block.type === "text" && typeof block.text === "string" ? [block.text] : [];
105+
}).join("\n"),
106+
})) }),
107+
});
108+
expect(response.ok).toBe(true);
109+
}
110+
writeFileSync(join(root, "requests.json"), JSON.stringify(requests, null, 2));
111+
const pids = JSON.parse(readFileSync(join(dataDir, "cortexkit", "rust-e2e-pids.json"), "utf8")).pids as { role: string; pid: number }[];
112+
for (const { role, pid } of [{ role: "bridge", pid: process.pid }, ...pids]) {
113+
const inventory = execFileSync("lsof", ["-nP", "-p", String(pid), "-Fn"], { encoding: "utf8", timeout: 10000 });
114+
writeFileSync(join(root, `lsof-${role}.txt`), inventory);
115+
const paths = inventory.split("\n").filter((line) => /^n.*\.(db|sqlite)(-(wal|shm))?$/.test(line)).map((line) => line.slice(1));
116+
expect(paths.every((path) => resolve(path).startsWith(`${taskRoot}/`) || realpathSync(path).startsWith(`${taskRoot}/`))).toBe(true);
117+
if (role !== "daemon") expect(paths.some((path) => path.endsWith("context.db"))).toBe(true);
118+
console.log(`${role} pid=${pid}: ${JSON.stringify(paths)}`);
119+
}
120+
console.log(`captured ${requests.length} fake provider requests; compartments=${JSON.stringify(db.prepare("SELECT COUNT(*) AS n FROM compartments WHERE session_id=?").get(session))}`);
121+
for (const request of requests) {
122+
expect(request.system).toBeUndefined();
123+
expect(request.messages[0].role).toBe("user");
124+
expect(request.messages[1].role).toBe("user");
125+
expect(request.messages[0].content).toContain("<project-memory>");
126+
expect(request.messages[0].content).toContain(memory);
127+
expect(request.messages[1].content).toContain("<session-history-since>");
128+
}
129+
expect(db.prepare("SELECT project_path FROM session_projects WHERE session_id=? AND harness='opencode2'").get(session)).toEqual({ project_path: identity });
130+
} finally {
131+
unregister();
132+
await adapter?.dispose();
133+
await stack?.stop();
134+
server.stop(true);
135+
db.close();
136+
}
137+
}, 60000);
138+
}

‎packages/plugin/src/features/magic-context/session-project-storage.ts‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
import { getHarness } from "../../shared/harness";
1+
import { getHarness, type HarnessId } from "../../shared/harness";
22
import type { Database, Statement as PreparedStatement } from "../../shared/sqlite";
33
import { isUserHomeDirectory } from "./memory/project-identity";
44

@@ -94,6 +94,7 @@ export function recordSessionProjectIdentity(
9494
db: Database,
9595
sessionId: string,
9696
projectPath: string | undefined,
97+
harness: HarnessId = getHarness(),
9798
): SessionProjectIdentityChange | undefined {
9899
if (!sessionId || !projectPath) return undefined;
99100
// A session started exactly at the user's home directory is not a project.
@@ -105,7 +106,6 @@ export function recordSessionProjectIdentity(
105106
isUserHomeDirectory(projectPath)
106107
)
107108
return;
108-
const harness = getHarness();
109109
const now = Date.now();
110110
let previousProjectPath: string | undefined;
111111
db.transaction(() => {

0 commit comments

Comments
 (0)