Add open-core readiness evidence slice

[KryptosAI]

Summary

This draft PR turns the open-core readiness plan into a focused, reproducible implementation slice.

What changed

• Added deterministic per-tool decisions to run artifacts:
  • high-severity tool findings → block
  • low/medium findings → review
  • no tool-level findings → allow
  • decisions retain the originating finding IDs and are shown in terminal and MCP output.
• Added the public benchmark corpus structure under benchmarks/ with safe, high-risk, and ambiguous seeded fixtures.
• Added docs/rules.md with evidence sources, severity/decision mapping, reproduction guidance, and limitations.
• Added a provider-neutral behavioral evaluation scaffold:
  • deterministic trace evaluation
  • allowed/forbidden tool assertions
  • result-pattern assertions
  • resource-scoped and global side-effect assertions
• Added docs/behavioral-evals.md and focused tests.
• Corrected documentation/code drift:
  • README MCP server count updated from 10 to the 13 tools currently registered.
  • removed the duplicate recordSessionStart call in the scan flow.
  • corrected the architecture description of the actual check pipeline.
• Centralized the pure scoring model in src/score-model.ts, shared by the CLI and hosted worker, removing scoring drift risk while preserving the worker's security-lite scope.
• Made optional seatbelt detection local-only so report rendering never starts network package resolution.

Validation

• npm test — full suite completed with no failing-test output after the worker-hardening expectation was updated for the shared scorer.
• npm run lint — passed.
• npm run typecheck — passed for the package and API worker.
• npm run build — passed.
• npm run validate:artifacts — passed for checked-in artifacts.
• npm pack --dry-run — readiness docs and benchmark assets included.
• Focused readiness tests: 22 passed; worker hardening tests: 3 passed.

Remaining gaps

• Receipt key generation/verification CLI commands remain tracked in issue Add receipt keygen and receipt verify commands #288.
• Permission-delta reporting still needs more precise fixtures and policy semantics; issue Track permission-delta rate for parser repair and MCP schema drift #150 remains the follow-up.
• The behavioral runner intentionally does not perform model-backed execution or live side effects yet.
• Broader CI/diff admission integration remains a follow-up under issue CI/CD marketplace: extend setup-ci to GitLab, CircleCI, Jenkins, BitBucket #221.

Open-core boundary

This PR keeps the scanner, evidence engine, artifact schemas, receipts, local baselines/diffs, public benchmark fixtures, and deterministic local eval runner open. Hosted fleet coordination, organization/team management, central policy administration, historical retention, scheduled rescans, hosted approvals, blast-radius analysis, enterprise IAM, managed orchestration, private indexes, compliance workflows, and hosted decision APIs remain proprietary control-plane work.

[github-actions]

🔭 MCP Observatory — Action needed: run did not clear the gate

Safety verdict

Blocked — One or more checks can break agent dependence and should be fixed before production use.
Top risks: startup: server failed to start

📊 Summary

Gate: fail · 0 tools · 0 prompts · 0 resources

Next action

Run the target command manually, compare stderr with the diagnosis below, and only raise timeoutMs if startup is genuinely slow.
Add CI: npx @kryptosai/mcp-observatory setup-ci --all --command "npx -y "

---

_{🔭 MCP Observatory — test your MCP servers for breaking changes · ⭐ Star}