feat!: Osiris v0.6.0 — conversation productization engine over cf-ng - #70
Open
padak wants to merge 31 commits into
Open
feat!: Osiris v0.6.0 — conversation productization engine over cf-ng#70padak wants to merge 31 commits into
padak wants to merge 31 commits into
Conversation
Rebuilds Osiris around cf-ng as the connector substrate: the engine becomes a relay that records an agent's exploratory conversation, freezes it into a fingerprinted plan, and runs it deterministically without an LLM. Key decisions: - cf-ng owns connectors and credentials; the engine owns evidence and determinism - host agent owns the LLM loop; the engine ships a skill, never an API key - recording relay (not post-hoc compilation) so evidence is ground truth - four pin classes with distinct drift policies instead of a binary claim - Docker as a packaging format, not an isolation boundary; E2B dropped - ~19k LOC of drivers, connectors, remote/ and chat deleted in the same initiative Depends on keboola/cf-ng#25, #26, #27.
Reverses the earlier decision to hold intermediate data in memory. ADR-0043's DuckDB data bus is retained: its design is the best-measured in the repo (~1.5M rows/s, 98% memory reduction) and only its wiring is broken, so fixing the shared RunContext becomes the first deliverable of phase 1. Adds section 4.5 on volume. cf-ng's read caps are env-configurable defaults, not ceilings; the binding constraint is the synchronous inline-payload shape of POST /tools/call, which raising a cap does not change. Engine-side pagination into DuckDB is the v1 mechanism; a cf-ng bulk read path (streaming, async job, or cf-ng#11 land-to-Storage) is recorded as a new dependency. Adds section 12 with worked examples: the authoring loop, cron/GitHub Actions/ Keboola orchestration hosts, and what a drift failure looks like.
12 tasks, 64 bite-sized TDD steps covering phase 0+1: clear the v0.5.4 tree, harvest the determinism core and filesystem contract, build the shared RunContext that v0.5.4 lacked, the cf-ng client with pin capture, the plan model and freeze, the runner with pin verification, the recording MCP relay, the CLI, and a round-trip guarantee test. Phases 2-5 get their own plans.
Deletes drivers, connectors, remote (E2B), mcp, runtime, core, cli, components, prompts, prototypes and tools along with their tests -- 362 tracked files, ~50k lines of production code and ~53k of tests. Harvested modules are re-created from the implementation plan rather than imported, so nothing depends on the deleted tree. Also prunes 62 stale ruff per-file-ignores pointing at deleted paths, which is what kept 'make lint' red after the deletion. BREAKING CHANGE: the v0.5.4 CLI and OML pipeline format are gone.
Harvested from v0.5.4 with require_fingerprint added, so verification has a caller that raises instead of a helper nobody invokes. v0.5.4 computed fingerprints faithfully and called verify_fingerprint nowhere at runtime.
Found during Task 2 execution: the plan's fingerprint_dict source fails make lint as written, because PL is in ruff's select and only tests/ and scripts/ carry a per-file ignore.
Pins come from GET /connectors/{id}/tools, not the MCP gateway, which
rewrites inputSchema to inject credentials and credentials_label -- a
gateway-derived hash would drift whenever a connector's credential
schema changed, even though the tool itself had not.
…reams Appends take flock(LOCK_EX) with write/flush/fsync inside the lock, so concurrent writers cannot interleave a partial line -- stress-verified at 20 trials x 128 records x 40KB payloads x 16 threads. Session redacts at write time and has no module-level current-session global; v0.5.4 had one with a comment admitting a thread-local would be better. The session is passed explicitly instead.
slugify collapses every traversal character, so path escape is structurally impossible rather than checked after the fact. The regex preserves underscores because run ids use them as structural separators (run_<ts>_<hex>); hyphenating them would decouple a run's log directory from the id recorded in the run index. The plan's path-escape test was vacuous -- Path.parents is lexical, so it passed for paths that genuinely resolve outside base_path. Rewritten to assert no '..' survives, resolved.is_relative_to(base), and no silent collapse onto base, across 6 hostile inputs x 5 path builders.
Replaces the two divergent inline contexts of v0.5.4, neither of which provided get_db_connection() while all seven drivers required it -- which is why v0.5.4 could not execute a pipeline at all. Characterizes DuckDB's locking, which constrains later phases: within one process a second context on the same path is an ALIAS (shared instance, no isolation), across processes the file lock is exclusive and raises. The lock releases cleanly on close. The data bus therefore has a single process owner.
Measured during Task 7: in-process second contexts are aliases with no isolation; cross-process opens are locked out and raise. Constrains the containerized runtime and any future per-step process split.
Every relayed call is recorded with its arguments, tool schema pin, outcome and duration, so freeze is grounded in ground truth rather than the agent's recollection. Schema pins come from the REST manifest and are cached per connector, so a two-tool connector costs one manifest fetch. The plan's MCP wiring targeted SDK v1 and does not run on the installed mcp 2.0.0: the @server.list_tools()/@server.call_tool() decorators no longer exist, handlers are constructor kwargs taking (ctx, params) and returning result models. Rewritten against the real SDK and covered by an in-memory client/server round trip.
Freeze validates every cfng_call against the live tool manifest, pins the tool contract and catalog version, and rejects literal secrets outright. Determinism is proven rather than assumed: the plan's own test can pass by coincidence when two freezes land in the same wall-clock second, so the test monkeypatches datetime to 2001 and 2050 and still requires an identical manifest hash. Verified again across two independent processes, which rules out dict-ordering and hash-randomization dependence.
…stalled 2.0.0 The decorator API in the plan does not exist in mcp 2.0.0. Points readers at the shipped osiris/relay/server.py as the correct reference.
The previous commit landed the lint-trap constraints but missed this warning because the function signature in the plan differs from what the patch script matched.
Pins are verified before the first tool call; contract drift aborts with nothing called. assert is first-class so a silent upstream change stops the run instead of producing an empty result. The plan's rows_to_arrow was broken three independent ways on DuckDB 1.5.5: the relation is built on the module default connection and cannot be registered on the run's connection; the resulting shape is a single JSON column named j, not the row's columns; and it holds the payload in memory twice. Replaced with NDJSON via read_json_auto(sample_size=-1), which also leaves a durable per-step artifact. sample_size=-1 is load-bearing: a sampled scan drops keys that first appear in late rows. Identifiers are quoted -- 'fetch' is a DuckDB reserved word, so the plan's own step id would not parse unquoted.
'fetch' is reserved in DuckDB 1.5.5, so the plan's own draft plans would raise ParserException. Affects Task 9's and Task 12's DRAFT.
…nce mismatch The plan minted a run id in the CLI for the evidence directory while Runner.execute() minted a second, different one that went into the ledger. A runs.jsonl row therefore named a run id matching no directory on disk -- the ledger could not lead you to its own evidence. The invocation id is now used consistently for ledger, directory and output. The CLI also verifies the manifest fingerprint against the fingerprints.json beside it before executing. Reading a manifest and ignoring the fingerprint file next to it is exactly the v0.5.4 failure this rebuild exists to end. Other corrections: config and credentials are checked before arguments and all missing env vars are reported at once; failed runs are recorded in the ledger with real timestamps; serve's banner goes to stderr because stdout carries JSON-RPC framing; evidence paths derive from Paths so plan metadata cannot escape the run directory. Adds nosec B608 to the three step modules: identifiers are quote_ident'd and the SQL body is the plan author's own, from an artifact that is fingerprint-verified before the run starts.
…ntom Freeze then run twice must produce identical evidence -- compared on the actual events.jsonl and metrics.jsonl with ts/run_id/duration stripped, not merely on summary row counts -- and a tampered manifest must be detected against the recorded fingerprint. The skip ban closes the hole that let v0.5.4 ship a runtime that could not execute anything: tests/integration/test_compile_run.py carried 'pytestmark = pytest.mark.skip(reason="Integration tests need rewrite for FilesystemContract v1 API")' -- a plausible-sounding reason that silenced the only real check. Verified to have teeth against both spellings. Makefile: 'make ci' was gating on 'make type-check', which only echoed a message. Removed it and every target naming a deleted subsystem.
…efuted Five independent skeptics attacked the implementation rather than reading the tests. Every guarantee the walking skeleton claims was broken, with reproducible evidence, while the full suite stayed green and all three lint gates passed.
Adversarial verification refuted every guarantee the walking skeleton claimed, while 139 tests passed and ruff/black/bandit were clean. Artifact integrity: only fingerprints[plan] was verified; [pins] and [manifest] were written and never read, so recomputing one hash via the repo's own public API let tampered SQL run at exit 0 -- and the ledger recorded the pre-tamper hash, read from the manifest's own unhashed self-declaration. Now all three are verified plus the internal relation manifest == sha256(plan + pins), the build directory name is cross-checked, and the ledger takes the verified value. Residual: an unkeyed checksum still yields to an attacker who rewrites every file and renames the directory; that needs a signature. Determinism: a set anywhere in a draft made the hash PYTHONHASHSEED dependent (8 processes, 8 hashes). Non-JSON types are now rejected at freeze, which also closes NaN/Infinity collapsing four distinct drafts into one hash. The determinism test now runs across processes, since the old one called freeze twice in one process and could not fail. Redaction: four live leaks -- runs.jsonl, NDJSON artifacts, the DuckDB file and stdout -- while the test greped only the one file already correct. redact() now walks dict keys, tuples, sets and bytes; rows are redacted before the artifact is written and the table is built from that file, so nothing enters DuckDB pages. Sweeps now byte-grep every file under base_path on both the success and failure paths, with a positive control proving the grep works. Drift: 'want.output is not None' meant adding an outputSchema after freeze was undetectable while removing one aborted. Comparison is now symmetric. Empty or partial pins on a plan with tool calls is a hard failure, and a pins_verified event distinguishes a verified run. Also: deleted scripts/ (12 files, all v0.5.4 residue, 5 crashing on import), fingerprint_dict and PathsConfigError (dead), dropped unused python-dotenv. The skip guard is now AST-based -- three common spellings evaded the regex while hiding failing tests.
…med sound Round-1 defects are closed. The skeptics went past them to different ones: foreign cfng_ tokens unredacted, a policy field that silently disables the abort while the evidence still claims verification, annotations excluded from the tool pin, ToolPin the one un-sealed model, and an mcp floor that cannot run the code. Notably the report also confirms what holds: determinism across process, cwd, TZ, locale, PYTHONHASHSEED and clock (5 interpreters, 3 fake clocks, 120 fuzzed drafts, zero variance and zero collisions), semantic sensitivity across 27 probes, 11 reformatting attacks correctly ignored, and abort-before-first-call real and covered.
…upports Two rounds of adversarial verification refuted every guarantee as originally worded. Most round-2 refutations were defects; several were the wording. Section 4.3.1 now gives each claim with its bound: - deterministic, but the hash covers the pins, so it names a plan-against-an-environment rather than a plan - tamper-evident against accident and partial edit, not against an attacker who rewrites every file -- an unkeyed checksum stored beside what it protects is not a signature - aborts on drift, conditional on a policy the plan author sets - pins verified at t0, not continuously - secret-free by known value plus credential-shaped pattern: mitigation, not proof Also records the lesson that outlived the specific bugs: a green test is a claim, not evidence. The round-1 leak test passed 5/5 while four leaks were live, because it greped the one file already correct.
…ret shapes
ToolPin was the one model left with pydantic's default extra='ignore'
while every sibling was extra='forbid'. 100KB and a live credential could
be smuggled into a pins.tools entry: dropped on load, never hashed, and
the artifact passed all five integrity checks. Sealed.
tool_pin did not hash annotations -- the MCP machine-readable safety
hints -- so flipping destructiveHint false to true was undetectable under
the default fail policy. And 'inputSchema or {}' collapsed absent, {},
null and false into two fingerprints, though {} means 'accepts anything'
and false means 'accepts nothing'. Both halves of the pin now agree.
NOTE: this changes every existing pin value; artifacts frozen earlier
must be re-frozen.
Drift evidence lied. Under policy 'ignore' the pins_verified event was
byte-identical to a clean run, and the ignored diff reached disk nowhere.
The event name is now the assertion: pins_verified only when nothing
drifted, pins_drift_suppressed otherwise, plus drift_ignored carrying the
diff.
Redaction was exact-substring against one env var, so a foreign cfng_
token -- including the credentials argument cf-ng injects into every tool
schema by design -- was written verbatim beside the process's own token
shown as ***. Added a prefix-anchored shape rule at the seam, applied
before exact-value matching so an explicit secret that is a substring of
a longer token cannot punch a readable hole in it. Deliberately not
entropy-based: evidence is full of long opaque strings that must survive.
freeze's own regex missed the real cfng_v1.<base64> shape and ran before
catalog_version was populated, so a cf-ng reflecting the credential put
it in the artifact. One shared pattern now, guard runs after every field
is populated. Refuses rather than masking: masked bytes would no longer
hash to the recorded fingerprint, and hashing the masked form would ship
an artifact that silently means something other than the draft.
mcp floor was >=1.2.1 while the code needs the 2.x Server API -- osiris
serve was dead on arrival on a legal resolution. Raised to >=2.0.0;
declared anyio and requests, which were used but undeclared.
Round 3 deliberately not run: adversarial verification does not converge, so the stopping point is fixing what is cheap and unambiguously wrong, then stating the guarantees at their real strength. Highest-leverage open item is that no CI job can fail a PR.
No required check ran the suite. research.yml was the only workflow invoking pytest and carried continue-on-error at job AND step level plus '|| true'; ci-mcp.yml, mcp-phase1-guards.yml, e2b-tests.yml and e2b-manual.yml were path-filtered on osiris/mcp, osiris/remote and tests/e2b, all deleted, so they could never trigger. Every command they ran was dead. Removed all five. ci.yml runs lint, security, tests and a package check, all blocking, on every pull request. The lint and security jobs mirror 'make lint' and 'make security' command for command -- if they diverge, the local gate lies. Tests run on 3.11 and 3.13. Testing the declared floor is not ceremony: the audit found mcp>=1.2.1 declared while the code needed the 2.x Server API, so 'osiris serve' was dead on arrival under a legal resolution. The package job builds a wheel and installs it into a fresh interpreter from its own declared dependencies, then exercises the CLI. The audit found the suite passing against a stale v0.5.4 virtualenv still carrying e2b, supabase, openai and pandas; only a clean install proves the package is self-contained. Verified locally end to end. detect-secrets: 'scan --baseline X' rewrites X and exits 0 whatever it finds -- the old workflow and the old make target both ran exactly that, so neither could ever fail. Both now run detect-secrets-hook, verified to pass clean and to reject a planted GitHub token. The baseline is regenerated (it still indexed components/ and tests/drivers/, deleted in the rebuild) and its 58 findings audited as non-secrets; all are documentation examples such as mysql://user:pass@localhost and sk-1234567890, none in osiris/ or tests/. Also removes the shopify.extractor docs example, a v0.5.4 driver skeleton whose pandas and requests imports were the last thing blocking a clean-venv run, and drops requests from the dev extras with it. CODEOWNERS and MANIFEST.in are rewritten; every path they named was deleted, and neither tool errors on a pattern that matches nothing.
GitHub push protection scans source text, not intent, and rejected the contiguous literal in these two synthetic fixtures. The alternative -- GitHub's 'allow this secret' link -- would whitelist a credential shape repo-wide to make a test pushable, which is the worse trade. Runtime values are unchanged, so the tests are unchanged.
…ath's parent Three items from the round-2 open list. .env loading: python-dotenv was declared and never imported, so a valid .env did nothing and 'osiris doctor' reported the variables unset -- a dependency shipped but not wired is a promise the CLI does not keep. Loads from the working directory only, with override=False so an exported variable still wins; walking up to find a .env would make the credential a command runs with depend on where you were standing. Verified from a clean wheel install. RunContext.secrets: cfng_call reached into ctx._session via getattr, a coupling that survives until someone renames the attribute and then fails silently by redacting nothing. The property returns a copy, so a step cannot empty the session's list. Integrity check 5 validated only the build directory's leaf name, so a verified artifact could be moved under a different plan name -- the layout would say one thing while the manifest said another, and the run ledger keys on the plan name. Mutation-checked: removing the parent check kills the new test. Also refreshes the package metadata, which still described v0.5.4 to PyPI as an 'LLM-first conversational ETL pipeline generator' keyworded with mysql, supabase, openai, claude and gemini.
…rrals Remaining open: keyed signing, per-call pin re-verification (TOCTOU), and unifying the pin-key format -- the last of which invalidates every frozen artifact, so it waits for a reason to pay that cost.
Found by hands-on testing, not by the suite. run_log_dir() ran the run id through slugify(), which lowercases. The run index therefore recorded run_20260810T192357Z_493c91 while the directory on disk was run_20260810t192357z_493c91. On macOS that is invisible -- APFS is case-insensitive by default -- but on Linux, which is CI and the deployment target, copying a run id out of runs.jsonl and looking for its evidence fails outright. An audit trail you cannot follow is not one. No test caught it because every test built both sides of the comparison through the same lowercasing helper. Two identically broken paths agree. slugify stays for human-authored names, where normalizing however someone typed it is the point. sanitize_segment is the case-preserving sibling for identifiers Osiris generated itself, with the same traversal defense: the character class is the only difference. Verified on a real case-sensitive volume, and mutation-checked -- restoring slugify(run_id) kills two tests.
…aceback Found while dry-running the live probe: pointing the client at a dead host produced an httpx.ConnectError stack. A caller should not have to know httpx exists to handle 'cf-ng is unreachable', and every caller inherited the gap -- the pin probe had been fixed for this one layer up, but the client below it still leaked. A non-JSON 200 is also an error now: something answering at that URL that is not cf-ng (a proxy, a captive portal, a login page) used to surface as a JSONDecodeError from inside the client. Both get synthetic negative statuses so stays one comparable field, and a so they are never quoted at a human -- 'cf-ng answered -1' is a number that exists only inside this module and reads as a bug rather than as 'the host did not answer'. Transport errors are retryable; a malformed response is not. Two existing tests encoded the old behaviour (asserting status is None, and the literal 'status 403'); both updated to assert the new contract rather than relaxed. Also makes isort honour .gitignore, which ruff already did: make lint disagreed with itself the moment a scratch file appeared in an ignored directory.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Rebuilds Osiris around cf-ng as the connector substrate. Same goal as v0.5.x — AI designs once, the runtime executes deterministically — new technology and new architecture.
484 files, +11,702 / −123,289. Production code goes from 48,700 lines to 2,448.
Why
You hold an exploratory conversation with an AI about your own environment — which leads are in Salesforce, which opportunity moved through PoC fastest, how is headcount developing. The conversation ends when you find the answer. Then you want to automate what it taught you, on a schedule, and for something running every 15 minutes you do not want an LLM: it is mostly unnecessary, non-deterministic, and costs money on every tick.
cf-ng makes ~700 connectors callable by an agent in real time. It deliberately stops there — verified by grep, not inferred: no run record (the vault schema has no runs key), no run id in responses, telemetry is one stdout line with no arguments/result/duration/outcome, no idempotency, no orchestration, no scheduling, no result persistence, no packaging, no deployment artifact.
This PR is the layer that turns a cf-ng conversation into a fingerprinted, replayable, explainable artifact.
Design:
docs/design/osiris-0.6.0-engine.md.Plan:
docs/superpowers/plans/2026-08-10-osiris-060-walking-skeleton.md.The honest starting position
v0.5.4 could not execute a pipeline.
RunnerV0.RunnerContextwas defined inline inside a method and exposed onlyoutput_dirandlog_metric, while all seven drivers calledctx.get_db_connection().ProxyWorker.SimpleContexthad the same gap. Local and E2B execution raisedAttributeErrorfor every migrated driver, and the integration tests that would have caught it werepytestmark = pytest.mark.skip(reason="Integration tests need rewrite for FilesystemContract v1 API")— a plausible-sounding reason that silenced the only real check.osiris compiledid work and was deterministic. That is the dividing line this PR harvests along: the compilation spine lives, the execution layer was dead.What it does
Three deployment units, none holding an LLM API key:
osiris servebuild/<hash>/osiris runserverelays your agent's cf-ng tool calls and records every one — arguments, result shape, tool schema hash, duration, outcome.freezevalidates a plan against both those recordings and cf-ng's live schemas, pins what it can, fingerprints, and emitsbuild/<hash>/.runverifies the pins before the first call and executes against cf-ng, passing data between steps through a per-run DuckDB file.The engine is in the path during exploration on purpose: its differentiator is evidence, and something not in the path cannot produce evidence — only accept claims.
Reviewing this
The diff is large but mostly deletion. Suggested order:
docs/design/osiris-0.6.0-engine.md— the design, especially §4.3.1, which states each guarantee with its bound.osiris/— 2,448 lines across 8 subpackages.docs/reports/2026-08-10-v060-adversarial-verification/— two rounds of adversarial verification and what remains open.Adversarial verification
The implementation was attacked twice by independent agents instructed to refute each guarantee by breaking the implementation rather than reading the tests.
Round 1 refuted all five, while 139 tests passed and ruff/black/bandit were clean:
fingerprints["plan"]was verified —[pins]and[manifest]were written and never read, so recomputing one hash with the repo's own public API let tampered SQL run at exit 0, and the ledger recorded the pre-tamper hashsetanywhere in a draft made the hashPYTHONHASHSEED-dependent: 8 processes, one draft, 8 hashesoutputSchemaafter freeze was undetectable; removing one aborted correctlyRound 2 refuted all five again, on different and deeper defects — foreign
cfng_tokens unredacted, a policy field that silently disabled the abort while the evidence still claimed verification,annotationsexcluded from the tool pin,ToolPinthe one un-sealed model, and anmcp>=1.2.1floor that could not run the code.Both rounds are fixed. Round 3 was deliberately not run: adversarial verification does not converge, so the stopping point is fixing what is cheap and unambiguously wrong, then stating the guarantees at the strength the implementation actually supports. Round 2 also confirmed what holds — determinism across process, cwd, TZ, locale,
PYTHONHASHSEEDand clock (5 interpreters, 3 fake clocks, 120 fuzzed drafts, zero variance and zero collisions), semantic sensitivity across 27 probes, 11 reformatting attacks correctly ignored, abort-before-first-call real and covered.Known bound, stated in the design: artifact integrity is an unkeyed checksum stored beside what it protects. Every partial edit is caught; an attacker who rewrites every file and renames the directory is not. That needs a signature.
CI
Five workflows are removed. None of them could fail a PR:
research.ymlwas the only one invoking pytest and carriedcontinue-on-errorat job and step level plus|| true; the other four were path-filtered onosiris/mcp,osiris/remoteandtests/e2b, all deleted, so they could never trigger.ci.ymlreplaces them with one blocking gate — lint, security, tests on 3.11 and 3.13, and a job that builds a wheel and installs it into a fresh interpreter from its own declared dependencies. That last one is not ceremony: the audit found the suite passing against a stale v0.5.4 virtualenv still carrying e2b, supabase, openai and pandas.Also:
detect-secrets scan --baseline Xrewrites X and exits 0 whatever it finds. Both the old workflow and the old make target ran exactly that, so neither could fail. Both now rundetect-secrets-hook, verified to pass clean and reject a planted GitHub token.Breaking changes
chat,compile,validate,logs,mcp,connections,componentsno longer exist; the CLI isinit,serve,freeze,run,doctor.osiris/v1Plan format.52b3f01will fail verification: addingannotationsto the tool pin changed every pin value.Depends on
502 "Upstream provider error."Not in this PR
Scheduling (cron/Actions/Keboola orchestration runs the artifact), the plugin/skill served by the engine, Docker packaging, AIOP export,
osiris replan, pagination for volume, artifact signing. Open items with cost estimates are inROUND-2.md.