From 86a0d7ed4137493beee0e2be1dd21a680b859bac Mon Sep 17 00:00:00 2001 From: Saiteja Kura Date: Mon, 13 Jul 2026 17:28:14 +0000 Subject: [PATCH 1/4] RFD: predicate caching Add design for caching predicate results using file-based watch paths. Two mechanisms: syntactic (watch field on skill groups) and runtime (stdout JSON from shell predicates). Coordinates with PM interface for depends-on() caching via Cargo.lock. --- md/rfds/predicate-caching/README.md | 81 +++++++++++++++++++++++++++++ 1 file changed, 81 insertions(+) create mode 100644 md/rfds/predicate-caching/README.md diff --git a/md/rfds/predicate-caching/README.md b/md/rfds/predicate-caching/README.md new file mode 100644 index 00000000..90032901 --- /dev/null +++ b/md/rfds/predicate-caching/README.md @@ -0,0 +1,81 @@ +# Predicate caching + +Predicates (especially `shell(...)`) spawn processes on every sync. This RFD helps them declare watch paths. Symposium caches results and skips re-evaluation while watched files are unchanged. No watch declaration = never cached. + +## Problem + +Auto-sync means predicates re-evaluate on every agent session start. A workspace with 10 plugins, each with a `shell(...)` gate, forks 10+ processes every time. + +## Design + +Two ways to declare watch paths: + +### 1. Syntactic + +For predicates whose inputs are known at authorship time: + +```toml +[[skills]] +predicates = ["shell(grep -q lambda CargoBrazil.toml)"] +watch = ["CargoBrazil.toml"] +source.path = "skills/lambda" +``` + +`watch` is a sibling field, not embedded in the predicate string. + +### 2. Runtime + +The shell command emits JSON to stdout: + +```json +{"result": true, "watch": ["CargoBrazil.toml"]} +``` + +Commands that print nothing or non-JSON fall back to exit-code semantics with no caching. Backward compatible. + +Runtime takes priority when both are present. Either is sufficient to enable caching. + +## How it works + +Fingerprint is `mtime + size` (same model as Cargo). A missing file is a valid fingerprint state — invalidates when the file appears. + +Cache lives at `predicate-cache.json`, keyed by the normalized predicate string. Two plugins with the same predicate share one entry. + +Algorithm: + +1. Look up predicate in cache +2. If found and all watched files match fingerprints → use cached result +3. Otherwise → evaluate, store result + watch paths if declared +4. On write, prune entries for predicates no longer in any manifest + +Cache is discarded on Symposium version upgrade. `watch = []` (empty) means static for the current invocation only — no persistent caching without at least one file to stat. + +## Built-in predicates + +- `workspace-member()` → cached, watch = [] (static per run) +- `path_exists(path)` → cached, watches the path itself +- `depends-on(name)` → cached, watches PM manifest (e.g. `Cargo.lock`) +- `env(...)` → never cached (env isn't file-observable) +- `shell(...)` → cached only with explicit watch declaration + +## PM integration + +`list_deps` return type expands to include optional watch paths: + +```rust +struct DepsResult { + ids: Vec, + watch: Option>, +} +``` + +`CargoPm` returns `[Cargo.lock]`. This is a return-type change, not a new method. + +## Implementation steps + +1. Extend `PredicateResult` with `watch: Option>`. All evaluators return `None` initially — no behavior change. +2. Cache storage: read/write, fingerprint comparison, dead-entry pruning. +3. Built-in predicates return their natural watch sets. +4. Shell runtime output: parse stdout JSON, fall back to exit code for non-JSON. +5. Syntactic `watch` field on `[[skills]]` groups. +6. PM `list_deps` watch support (coordinate with pm-split). From d699619082f12d935a8848a069df749c2b018225 Mon Sep 17 00:00:00 2001 From: Saiteja Kura Date: Tue, 14 Jul 2026 17:41:27 +0000 Subject: [PATCH 2/4] refactor: predicate caching RFD v2 --- md/SUMMARY.md | 2 + md/rfds/predicate-caching/README.md | 75 ++++++++++++++++------------- 2 files changed, 44 insertions(+), 33 deletions(-) diff --git a/md/SUMMARY.md b/md/SUMMARY.md index 59d4a66e..c3ed94e3 100644 --- a/md/SUMMARY.md +++ b/md/SUMMARY.md @@ -86,6 +86,8 @@ - [Template](./rfds/TEMPLATE/README.md) - [Accepted](./rfds/accepted.md) - [Registry-centric plugin distribution](./rfds/registry-centric-plugins/README.md) + - [Predicate caching](./rfds/predicate-caching/README.md) - [Completed](./rfds/completed.md) - [Configuration parsing and normalization](./rfds/config-normalization/README.md) - [RFD Process](./rfds/rfd-process/README.md) + - diff --git a/md/rfds/predicate-caching/README.md b/md/rfds/predicate-caching/README.md index 90032901..23b853f7 100644 --- a/md/rfds/predicate-caching/README.md +++ b/md/rfds/predicate-caching/README.md @@ -1,62 +1,72 @@ # Predicate caching -Predicates (especially `shell(...)`) spawn processes on every sync. This RFD helps them declare watch paths. Symposium caches results and skips re-evaluation while watched files are unchanged. No watch declaration = never cached. +## TL;DR + +Predicates (especially custom predicates) spawn processes on every sync. This RFD lets them emit watch declarations via JSONL events. Symposium caches results and skips re-evaluation while watched files/env vars are unchanged. No watch declaration = never cached. ## Problem -Auto-sync means predicates re-evaluate on every agent session start. A workspace with 10 plugins, each with a `shell(...)` gate, forks 10+ processes every time. +Auto-sync means predicates re-evaluate on every agent session start. A workspace with 10 plugins, each with a custom predicate, forks 10+ processes every time. ## Design -Two ways to declare watch paths: +Custom predicates already emit JSONL events to stdout. We add a `Watch` event variant: -### 1. Syntactic +```rust +#[derive(Serialize, Deserialize)] +enum CustomPredicateEvent { + // ... existing variants (e.g. SelectedCrates) ... + Watch { + files: Vec, + env: HashMap, + } +} +``` -For predicates whose inputs are known at authorship time: +Emitted as a JSONL line: -```toml -[[skills]] -predicates = ["shell(grep -q lambda CargoBrazil.toml)"] -watch = ["CargoBrazil.toml"] -source.path = "skills/lambda" +```jsonl +{"watch": {"files": ["CargoBrazil.toml"], "env": {"LAMBDA_ENV": "prod"}}} ``` -`watch` is a sibling field, not embedded in the predicate string. +The predicate result still comes from exit status — the watch event is just a cache hint. Predicates that don't emit a `Watch` event are never cached (current behavior preserved). -### 2. Runtime +Both `files` and `env` are optional (either alone is sufficient to enable caching). Files are relative to workspace root. -The shell command emits JSON to stdout: +## SDK helper -```json -{"result": true, "watch": ["CargoBrazil.toml"]} -``` +The `symposium-sdk` crate provides a helper that reads env vars and automatically emits the cache line: -Commands that print nothing or non-JSON fall back to exit-code semantics with no caching. Backward compatible. +```rust +// In a custom predicate binary: +let val = symposium_sdk::env::var("LAMBDA_ENV")?; +// ^ reads the var AND emits {"watch": {"env": {"LAMBDA_ENV": "prod"}}} +``` -Runtime takes priority when both are present. Either is sufficient to enable caching. +This way predicate authors get caching for free when they use the SDK. ## How it works -Fingerprint is `mtime + size` (same model as Cargo). A missing file is a valid fingerprint state — invalidates when the file appears. +Fingerprint for files is `mtime + size` (same model as Cargo). A missing file is a valid fingerprint state — invalidates when the file appears. Env fingerprint is the literal string value (or absent). -Cache lives at `predicate-cache.json`, keyed by the normalized predicate string. Two plugins with the same predicate share one entry. +Cache lives at `~/.symposium/cache/predicates.json`, keyed by the normalized predicate string. No workspace-root keying needed — watch paths are resolved relative to the current workspace root at stat time, so different workspaces naturally produce different fingerprints. Algorithm: 1. Look up predicate in cache -2. If found and all watched files match fingerprints → use cached result -3. Otherwise → evaluate, store result + watch paths if declared +2. If found and all watched files + env vars match fingerprints → use cached result +3. Otherwise → evaluate, store result + watch if declared 4. On write, prune entries for predicates no longer in any manifest -Cache is discarded on Symposium version upgrade. `watch = []` (empty) means static for the current invocation only — no persistent caching without at least one file to stat. +Cache is discarded on Symposium version upgrade. ## Built-in predicates -- `workspace-member()` → cached, watch = [] (static per run) +- `workspace-member()` → no cache needed (already cheap, in-memory) - `path_exists(path)` → cached, watches the path itself - `depends-on(name)` → cached, watches PM manifest (e.g. `Cargo.lock`) -- `env(...)` → never cached (env isn't file-observable) -- `shell(...)` → cached only with explicit watch declaration +- `env(FOO=BAR)` → cached, watches env value of `FOO` +- `shell(cmd)` / custom predicates → cached only when they emit a `Watch` event ## PM integration @@ -65,7 +75,7 @@ Cache is discarded on Symposium version upgrade. `watch = []` (empty) means stat ```rust struct DepsResult { ids: Vec, - watch: Option>, + watch: Option>, } ``` @@ -73,9 +83,8 @@ struct DepsResult { ## Implementation steps -1. Extend `PredicateResult` with `watch: Option>`. All evaluators return `None` initially — no behavior change. -2. Cache storage: read/write, fingerprint comparison, dead-entry pruning. -3. Built-in predicates return their natural watch sets. -4. Shell runtime output: parse stdout JSON, fall back to exit code for non-JSON. -5. Syntactic `watch` field on `[[skills]]` groups. -6. PM `list_deps` watch support (coordinate with pm-split). +1. Add `Watch` variant to `CustomPredicateEvent`. Parse it from predicate stdout. No caching yet — just capture the data. +2. Cache storage in `~/.symposium/cache/predicates.json`: read/write, fingerprint comparison, dead-entry pruning. +3. Wire built-in predicates: `path_exists` emits file watch, `env` emits env watch, `depends-on` uses PM watch paths. +4. Add `symposium_sdk::env::var()` helper that auto-emits the cache event. +5. PM `list_deps` watch support (coordinate with pm-split). From 21fbff9b92cd892326b91a000e06d98f53318341 Mon Sep 17 00:00:00 2001 From: Saiteja Kura Date: Fri, 17 Jul 2026 21:27:15 +0000 Subject: [PATCH 3/4] refactor: predicate caching RFD v3 --- md/SUMMARY.md | 1 - md/rfds/predicate-caching/README.md | 75 ++++++++++++++--------------- 2 files changed, 35 insertions(+), 41 deletions(-) diff --git a/md/SUMMARY.md b/md/SUMMARY.md index c3ed94e3..80e45bc8 100644 --- a/md/SUMMARY.md +++ b/md/SUMMARY.md @@ -90,4 +90,3 @@ - [Completed](./rfds/completed.md) - [Configuration parsing and normalization](./rfds/config-normalization/README.md) - [RFD Process](./rfds/rfd-process/README.md) - - diff --git a/md/rfds/predicate-caching/README.md b/md/rfds/predicate-caching/README.md index 23b853f7..57b85e5f 100644 --- a/md/rfds/predicate-caching/README.md +++ b/md/rfds/predicate-caching/README.md @@ -2,7 +2,7 @@ ## TL;DR -Predicates (especially custom predicates) spawn processes on every sync. This RFD lets them emit watch declarations via JSONL events. Symposium caches results and skips re-evaluation while watched files/env vars are unchanged. No watch declaration = never cached. +Predicates, especially custom predicates, spawn processes on every sync. This RFD lets custom predicates emit `Watch` JSONL events for files and environment variables. Symposium caches their results and skips reevaluation while the union of watched inputs is unchanged. No watch hints means cached indefinitely; `Volatile` means never cached. ## Problem @@ -10,81 +10,76 @@ Auto-sync means predicates re-evaluate on every agent session start. A workspace ## Design -Custom predicates already emit JSONL events to stdout. We add a `Watch` event variant: +Custom predicates already emit JSONL events to stdout. We add two variants: ```rust #[derive(Serialize, Deserialize)] enum CustomPredicateEvent { - // ... existing variants (e.g. SelectedCrates) ... + // ... existing variants ... Watch { files: Vec, env: HashMap, - } + }, + Volatile {}, } ``` -Emitted as a JSONL line: +A predicate can emit multiple watch hints: ```jsonl -{"watch": {"files": ["CargoBrazil.toml"], "env": {"LAMBDA_ENV": "prod"}}} +{"watch": {"files": ["CargoBrazil.toml"]}} +{"watch": {"files": ["Config"]}} +{"watch": {"env": {"LAMBDA_ENV": "prod"}}} ``` -The predicate result still comes from exit status — the watch event is just a cache hint. Predicates that don't emit a `Watch` event are never cached (current behavior preserved). +Symposium unions these into one watch set. A change to any watched input causes one reevaluation. Files are relative to the workspace root. -Both `files` and `env` are optional (either alone is sufficient to enable caching). Files are relative to workspace root. +The process exit status determines the predicate result. `Watch` and `Volatile` events only control caching. No `Watch` events means the result is cached indefinitely and predicates must report every changing input or Symposium may reuse a stale result. `Volatile` disables caching and takes precedence over watch hints. + +```jsonl +{"volatile": {}} +``` ## SDK helper -The `symposium-sdk` crate provides a helper that reads env vars and automatically emits the cache line: +The `symposium-sdk` crate provides a helper that reads an environment variable and emits its watch event: ```rust -// In a custom predicate binary: let val = symposium_sdk::env::var("LAMBDA_ENV")?; -// ^ reads the var AND emits {"watch": {"env": {"LAMBDA_ENV": "prod"}}} +// Emits {"watch": {"env": {"LAMBDA_ENV": "prod"}}}. ``` -This way predicate authors get caching for free when they use the SDK. +Multiple helper calls emit multiple events, which Symposium unions. ## How it works -Fingerprint for files is `mtime + size` (same model as Cargo). A missing file is a valid fingerprint state — invalidates when the file appears. Env fingerprint is the literal string value (or absent). +File fingerprints use `mtime + size`; missing is a valid state. Environment fingerprints use the current value or absent state. -Cache lives at `~/.symposium/cache/predicates.json`, keyed by the normalized predicate string. No workspace-root keying needed — watch paths are resolved relative to the current workspace root at stat time, so different workspaces naturally produce different fingerprints. +Cache lives at `~/.symposium/cache/predicates.json`. -Algorithm: - -1. Look up predicate in cache -2. If found and all watched files + env vars match fingerprints → use cached result -3. Otherwise → evaluate, store result + watch if declared -4. On write, prune entries for predicates no longer in any manifest +1. Look up the predicate in the cache. +2. If all watched inputs match, use the cached result. An empty watch set always matches. +3. Otherwise, evaluate the predicate and obtain its result from the exit status. +4. If it emitted `Volatile`, do not cache the result. Otherwise, store the result with the union of its watch hints. Cache is discarded on Symposium version upgrade. ## Built-in predicates -- `workspace-member()` → no cache needed (already cheap, in-memory) -- `path_exists(path)` → cached, watches the path itself -- `depends-on(name)` → cached, watches PM manifest (e.g. `Cargo.lock`) -- `env(FOO=BAR)` → cached, watches env value of `FOO` -- `shell(cmd)` / custom predicates → cached only when they emit a `Watch` event +- `workspace-member()` requires no cache because it is already cheap and evaluated in memory. +- `path_exists(path)` watches the path itself. +- `env(FOO=BAR)` watches the value of `FOO`. +- `shell(cmd)` is volatile because its inputs are unknown. +- Caching `depends-on(name)` is deferred to the PM interface work. ## PM integration -`list_deps` return type expands to include optional watch paths: - -```rust -struct DepsResult { - ids: Vec, - watch: Option>, -} -``` - -`CargoPm` returns `[Cargo.lock]`. This is a return-type change, not a new method. +Changes to `list_deps` and PM-derived caching are deferred to the PM interface work. ## Implementation steps -1. Add `Watch` variant to `CustomPredicateEvent`. Parse it from predicate stdout. No caching yet — just capture the data. -2. Cache storage in `~/.symposium/cache/predicates.json`: read/write, fingerprint comparison, dead-entry pruning. -3. Wire built-in predicates: `path_exists` emits file watch, `env` emits env watch, `depends-on` uses PM watch paths. -4. Add `symposium_sdk::env::var()` helper that auto-emits the cache event. -5. PM `list_deps` watch support (coordinate with pm-split). +1. Add and parse `Watch` and `Volatile` events while keeping the exit status as the predicate result. +2. Union watch hints, cache an empty watch set indefinitely, and give `Volatile` precedence. +3. Add cache storage and fingerprint comparison. +4. Wire `path_exists`, `env`, and volatile `shell` behavior. +5. Add `symposium_sdk::env::var()` to emit environment watch events. From f6b82347be682d2122c74b70ec425f0cdedc5169 Mon Sep 17 00:00:00 2001 From: Saiteja Kura Date: Wed, 22 Jul 2026 18:29:47 +0000 Subject: [PATCH 4/4] refactor: predicate caching RFD v4 switch to granular events on a non_exhaustive enum, and replace Volatile with WatchTime. --- md/rfds/predicate-caching/README.md | 56 ++++++++++++++--------------- 1 file changed, 27 insertions(+), 29 deletions(-) diff --git a/md/rfds/predicate-caching/README.md b/md/rfds/predicate-caching/README.md index 57b85e5f..a27ef707 100644 --- a/md/rfds/predicate-caching/README.md +++ b/md/rfds/predicate-caching/README.md @@ -2,7 +2,7 @@ ## TL;DR -Predicates, especially custom predicates, spawn processes on every sync. This RFD lets custom predicates emit `Watch` JSONL events for files and environment variables. Symposium caches their results and skips reevaluation while the union of watched inputs is unchanged. No watch hints means cached indefinitely; `Volatile` means never cached. +Predicates, especially custom predicates, spawn processes on every sync. This RFD lets custom predicates emit granular JSONL watch events for files, environment variables, and time. Symposium caches their results and skips reevaluation while every watched input is unchanged and no watch expires. No watch events means cached indefinitely; `WatchTime(0)` means never cached. ## Problem @@ -10,35 +10,33 @@ Auto-sync means predicates re-evaluate on every agent session start. A workspace ## Design -Custom predicates already emit JSONL events to stdout. We add two variants: +Custom predicates already emit JSONL events to stdout. We add one event per watched resource: ```rust -#[derive(Serialize, Deserialize)] +#[non_exhaustive] enum CustomPredicateEvent { // ... existing variants ... - Watch { - files: Vec, - env: HashMap, - }, - Volatile {}, + /// Result depends on the contents of the given file. + WatchFile(PathBuf), + /// Result depends on the value of the given environment variable. + WatchEnv(String), + /// Result becomes stale after this many milliseconds. + WatchTime(usize), } ``` -A predicate can emit multiple watch hints: +A predicate can emit any number of these events: ```jsonl -{"watch": {"files": ["CargoBrazil.toml"]}} -{"watch": {"files": ["Config"]}} -{"watch": {"env": {"LAMBDA_ENV": "prod"}}} +{"watchFile": "CargoBrazil.toml"} +{"watchFile": "Config"} +{"watchEnv": "LAMBDA_ENV"} +{"watchTime": 60000} ``` -Symposium unions these into one watch set. A change to any watched input causes one reevaluation. Files are relative to the workspace root. +Symposium unions the file and environment events. A change to any watched input or expiry of the shortest `WatchTime` causes one reevaluation. Files are relative to the workspace root. -The process exit status determines the predicate result. `Watch` and `Volatile` events only control caching. No `Watch` events means the result is cached indefinitely and predicates must report every changing input or Symposium may reuse a stale result. `Volatile` disables caching and takes precedence over watch hints. - -```jsonl -{"volatile": {}} -``` +The process exit status determines the predicate result. Watch events only control caching. No watch events means the result is cached indefinitely; predicates must report every changing input or Symposium may reuse a stale result. `WatchTime(0)` means the result is stale immediately, which effectively disables caching. The `#[non_exhaustive]` attribute leaves room for new watch kinds. ## SDK helper @@ -46,30 +44,30 @@ The `symposium-sdk` crate provides a helper that reads an environment variable a ```rust let val = symposium_sdk::env::var("LAMBDA_ENV")?; -// Emits {"watch": {"env": {"LAMBDA_ENV": "prod"}}}. +// Emits {"watchEnv": "LAMBDA_ENV"}. ``` Multiple helper calls emit multiple events, which Symposium unions. ## How it works -File fingerprints use `mtime + size`; missing is a valid state. Environment fingerprints use the current value or absent state. +File fingerprints use `mtime + size`; missing is a valid state. Environment fingerprints use the current value or absent state. Time fingerprints use the wall-clock time at which the entry becomes stale. Cache lives at `~/.symposium/cache/predicates.json`. 1. Look up the predicate in the cache. -2. If all watched inputs match, use the cached result. An empty watch set always matches. +2. If all watched inputs match and no `WatchTime` has elapsed, use the cached result. An empty watch set always matches. 3. Otherwise, evaluate the predicate and obtain its result from the exit status. -4. If it emitted `Volatile`, do not cache the result. Otherwise, store the result with the union of its watch hints. +4. Store the result with its emitted watch events. `WatchTime(0)` yields an immediately stale entry. Cache is discarded on Symposium version upgrade. ## Built-in predicates - `workspace-member()` requires no cache because it is already cheap and evaluated in memory. -- `path_exists(path)` watches the path itself. -- `env(FOO=BAR)` watches the value of `FOO`. -- `shell(cmd)` is volatile because its inputs are unknown. +- `path_exists(path)` emits `WatchFile(path)`. +- `env(FOO=BAR)` emits `WatchEnv("FOO")`. +- `shell(cmd)` emits `WatchTime(0)` because its inputs are unknown. - Caching `depends-on(name)` is deferred to the PM interface work. ## PM integration @@ -78,8 +76,8 @@ Changes to `list_deps` and PM-derived caching are deferred to the PM interface w ## Implementation steps -1. Add and parse `Watch` and `Volatile` events while keeping the exit status as the predicate result. -2. Union watch hints, cache an empty watch set indefinitely, and give `Volatile` precedence. -3. Add cache storage and fingerprint comparison. -4. Wire `path_exists`, `env`, and volatile `shell` behavior. +1. Add and parse `WatchFile`, `WatchEnv`, and `WatchTime` events while keeping the exit status as the predicate result. +2. Union watch events, cache an empty watch set indefinitely, and treat `WatchTime(0)` as immediate staleness. +3. Add cache storage and fingerprint comparison for files, environment variables, and expiry times. +4. Wire `path_exists`, `env`, and `shell` to emit their watch events. 5. Add `symposium_sdk::env::var()` to emit environment watch events.