The privacy layer for AI agent tool calls and Model Context Protocol (MCP).
Most PII masking libraries were designed for static document redaction, not autonomous AI agents. Replacing sensitive data with [REDACTED] or <EMAIL_1> completely breaks tool-calling workflows:
- Schema & Validator Rejection: When your AI agent attempts to invoke a Stripe API, PostgreSQL tool, or CRM integration, strict downstream validators (Pydantic, regex schemas, database constraints) immediately crash when handed
[REDACTED_CARD]instead of a 16-digit numeric card string. - One-Way Irreversibility: Once data is destroyed by one-way redaction, local tools cannot execute real actions. You cannot process a refund, send an email, or dispatch an SMS with
[EMAIL]. Bi-directional pseudonymization securely restores real PII at the tool boundary and re-masks outputs before returning to the model. - SSE Streaming Token Tearing: Naive regex replacements leak torn fragments when modern LLMs stream tokens across Server-Sent Events (SSE) chunks (e.g. streaming
"u_8f", then"9a1@shadow", then".vault").shadow-vaultprovides a lookahead holdback buffer that reassembles split tokens seamlessly without token tearing.
+---------------------------------------+
| END USER / CLIENT |
+---------------------------------------+
| ^
1. Raw User PII | | 8. Hydrated Real PII
(e.g. Card) v | (StreamHydrator SSE)
+---------------------------------------+
| SHADOW-VAULT PROXY |
+---------------------------------------+
| ^
2. Dehydrated | | 7. Dehydrated Tool
Safe Prompt v | Return Value
+------------------------------------+ +------------------------------------+
| CLOUD LLM / AGENT | | LOCAL MCP TOOL |
| (OpenAI / Anthropic / LangChain) | | (Stripe / SQL / API) |
+------------------------------------+ +------------------------------------+
| ^
| 3. Model Tool Call with | 4. @vault_guard
| Synthetic Tokens | Reversible Hydration
+-------------------------------------------------+
Install the core library (zero heavy NLP dependencies):
pip install shadow-vaultProtect any Python function or agent tool in seconds:
from shadow_vault import ShadowVault, vault_guard
vault = ShadowVault()
# 1. Protect your internal tool: automatically receives real PII, returns masked PII
@vault_guard(vault=vault, session_id_arg="session_id")
def charge_customer(email: str, card: str, session_id: str = "default") -> dict:
# Function executes with REAL PII: email='alice@example.com', card='4532 0151 1283 0366'
return {"status": "PAID", "receipt": email, "backup_phone": "+91 9876543210"}
# 2. Dehydrate safe prompt for the LLM:
prompt = vault.dehydrate("Charge alice@example.com on card 4532 0151 1283 0366", session_id="s1")
# Prompt: "Charge u_c09753@shadow.vault on card 4000009995970590"
# 3. Model executes tool using synthetic tokens -> returns sanitized output back to agent state:
result = charge_customer(email="u_c09753@shadow.vault", card="4000009995970590", session_id="s1")
# Result: {'status': 'PAID', 'receipt': 'u_c09753@shadow.vault', 'backup_phone': '+919000597059'}
# 4. Ephemeral Agent Runs (Auto-purges memory on exit):
with vault.session("run_123", auto_purge=True) as sess:
safe_prompt = sess.dehydrate("Refund alice@example.com on card 4532 0151 1283 0366")
# Run agent tools...
# 100% of mapping records are wiped from RAM/SQLite upon exiting the block!
# 5. Prevent Log Leaks with Drop-In Logging Filter:
import logging
from shadow_vault import VaultLogFilter
logging.getLogger().addFilter(VaultLogFilter(vault))
# Accidental logger.info(real_card) statements are scrubbed before reaching log sinks!Run the self-contained walkthroughs and cryptographic audits (requires NO API keys and NO external dependencies):
# 1. Independent Security & Cryptographic Self-Audit (6 automated checks):
shadow-vault verify
# or: python -m shadow_vault.verify
# 2. End-to-End Real Agent Zero-Leak Wire-Inspection Proof:
python examples/verify_agent_zero_leak.py
# 3. Autonomous Model Context Protocol (MCP) Stdio Gateway & AES-256 Vault:
python examples/demo_mcp_gateway.py| Resource | Description | Link |
|---|---|---|
| Interactive Google Colab Quickstart | Zero-install interactive walkthrough of dehydration, @vault_guard, streaming holdback buffer, and audit checks |
|
| Claude Desktop MCP Guide | Configure Anthropic's Claude Desktop application (claude_desktop_config.json) with shadow-vault mcp-proxy |
Read Cookbook |
| LangGraph Customer Support Agent | StateGraph multi-turn billing & dispute agent ensuring 0% raw PII in graph checkpoints or memory | Run Script |
| Agent Zero-Leak Wire Verification | Automated 5-assertion proof that 0% sensitive values touch external networks | Run Script |
Experience the 3-panel visual privacy playground directly in your browser with real-time token highlighting and sub-millisecond latency counters:
🌐 Live Demo (Zero Install): https://dhirajparihar.github.io/shadow-vault/
- Panel 1 (User & Local MCP Tool): Displays raw prompt with real PII and verifies
@vault_guardrestoring real values before local tools execute. - Panel 2 (What Cloud LLMs See): Highlights format-preserving synthetic
@shadow.vaulttokens in green over the wire. - Panel 3 (Live SQLite Vault): Inspects the live in-memory mapping table, BLAKE2b deduplication, and sub-millisecond latency counters.
- Pre-Loaded Scenarios: Includes Fintech UPI/PAN Refund Agent, Healthcare Patient Booking Agent, and DevOps API Key Log Analyzer.
shadow-vault uses zero-dependency regexes coupled with strict mathematical checksum algorithms so random numbers are never falsely flagged:
| Entity Type | Mathematical Checksum / Algorithm | Real Input Example | Format-Preserving Synthetic Output |
|---|---|---|---|
CREDIT_CARD |
Luhn Checksum (mod 10) (13–19 digits) | 4532 0151 1283 0366 |
4000009995970590 (passes Luhn!)
|
AADHAAR |
Verhoeff Dihedral |
3675 9834 6016 |
999905970594 (passes Verhoeff!)
|
PAN_CARD |
Indian Income Tax Format ([A-Z]{5}[0-9]{4}[A-Z]{1}) |
ABCDE1234F |
SHDWV7059Z |
UPI_ID |
Evaluated before email (@okaxis, @okhdfcbank, @paytm, etc.) |
dhiraj@okaxis |
u_a81c25@okshadow |
EMAIL |
RFC-5322 (ignores @shadow.vault & UPI handles) |
alice@example.com |
u_c09753@shadow.vault |
PHONE |
E.164, Indian mobile (+91 [6-9]\d{9}), US/NANP |
+91 9876543210 |
+919000597059 (valid 10-digit mobile)
|
LOCATION / ADDRESS |
Street suffix pattern matching (St, Ave, Blvd, etc.) |
742 Evergreen Terrace |
452 Shadow Vault Way |
SSN |
US Social Security Number (Area/Group/Serial checks) | 219-45-7890 |
900-99-7059 (unissued area code)
|
API_KEY |
OpenAI sk-..., GitHub ghp_..., Google AIza..., Bearer |
sk-abcdef1234567890abcdef... |
sk-sv-shadow-c0975300000000000000 |
💡 For Unstructured Names & Ambiguous Free-Text Entities: ShadowVault provides a first-class Microsoft Presidio NER adapter (
from shadow_vault.integrations.presidio import PresidioDetector). It pairs high-precision spaCy / transformer NER models with ShadowVault's format-preserving token synthesis and@vault_guardtool boundary hydration.
| Architectural Capability | shadow-vault |
Rizzo PII | Microsoft Presidio | LangChain PIIMiddleware |
|---|---|---|---|---|
| Primary Focus | AI Agent Tools & MCP | Document Anonymization (Italian Legal) | Document Redaction / NLP Extraction | Simple Prompt Pre-filtering |
| Format-Preserving Tokens | Yes (Pydantic & regex valid) | No (replaces with <LOC>, <PER>) |
No (replaces with <EMAIL>) |
No (replaces with [REDACTED]) |
| Reversible Tool Boundaries | Yes (auto-hydrates at tool) | No (one-way document masking) | Manual (requires custom glue) | No (one-way irreversible) |
| Model Context Protocol (MCP) | Yes (stdio JSON-RPC gateway) | No | No | No |
| SSE Stream Safe (No Tearing) | Yes (lookahead holdback buffer) | No | No | No |
| Zero-ML Core Footprint | Yes (~0 MB, pure Python) | No (trained local model weights) | No (~500MB spaCy / PyTorch) | Yes |
| At-Rest Database Encryption | AES-256-GCM + Blind Index | SQLite / Local File Plaintext | External DB Dependent | None |
| Fail-Closed Policy Controls | Yes (REDACT / RAISE / BLOCK) | No | No | No |
Positioning Note: If you are anonymizing static PDF/Word legal contracts in Italian, use Rizzo PII. If you need an enterprise NLP server for bulk text extraction, use Microsoft Presidio. If you are running autonomous AI agents (LangGraph, CrewAI, Claude Desktop) that make downstream tool calls and cannot tolerate broken schemas or leaked credentials, use ShadowVault.
from langchain_core.tools import tool
from langchain_openai import ChatOpenAI
from shadow_vault import ShadowVault, protect_tool, ShadowVaultChatWrapper
vault = ShadowVault()
session_id = "agent_session"
# Protect Tools: Inputs hydrated before execution, outputs dehydrated before returning
@tool
def lookup_db(card_number: str) -> dict:
"""Fetch record from database."""
return {"card": card_number, "balance": 1500}
safe_tool = protect_tool(lookup_db, vault=vault, session_id=session_id)
# Protect ChatModel: Prompts dehydrated before cloud LLM, tool-call arguments kept synthetic
llm = ChatOpenAI(model="gpt-4o")
safe_llm = ShadowVaultChatWrapper(llm=llm, vault=vault, session_id=session_id)Wrap any existing MCP server (Node.js, Python, Go, Rust) with a zero-code transparent privacy gateway:
# Sits between your Agent/Claude Desktop and downstream MCP servers over stdio:
shadow-vault mcp-proxy --session-id user_42 -- uvx mcp-server-sqlite --db /path/to/data.sqliteClaude Desktop / Cursor Configuration (claude_desktop_config.json):
{
"mcpServers": {
"secure-sqlite": {
"command": "shadow-vault",
"args": [
"mcp-proxy",
"--session-id", "claude_session",
"--fail-closed",
"--",
"uvx", "mcp-server-sqlite", "--db", "/path/to/database.db"
]
}
}
}Or intercept programmatic JSON-RPC messages in Python:
from shadow_vault import ShadowVault, MCPVaultInterceptor
vault = ShadowVault()
interceptor = MCPVaultInterceptor(vault=vault)
# 1. Intercept incoming tool call from model (restores real PII before tool executes):
safe_request = interceptor.intercept_request(mcp_request, session_id="user_session")
# 2. Intercept outgoing tool response before it reaches LLM (re-dehydrates result):
safe_response = interceptor.intercept_response(mcp_response, session_id="user_session")Unlike naive caches that store raw sensitive values in plaintext SQLite, shadow-vault provides authenticated envelope encryption at rest and cryptographic blind indexing:
from shadow_vault import ShadowVault, VaultStore
# Enable AES-256-GCM / authenticated encryption at rest:
vault = ShadowVault(
db_path="./vault.db",
encryption_key="my-master-vault-encryption-key", # Or via SHADOW_VAULT_KEY env var
)-
Zero Plaintext on Disk: All
real_valentries written to SQLite are authenticated ciphertexts (enc:gcm:v1:...orenc:std:v1:...). Direct inspection of.dbfiles reveals zero real PII. -
HMAC-SHA256 Blind Indexing: Fast
$O(1)$ database lookups without leaking plaintext or search patterns to anyone with read access to the database. - Tenant & Session Salting: Support for custom salts prevents cross-session rainbow table correlation.
Different agent tools require different privacy boundaries. A Slack notification tool should never receive raw credit cards, while a Stripe tool needs card details but not Aadhaar or SSNs:
from shadow_vault import ShadowVault, ToolPolicy, FailClosedAction, vault_guard
vault = ShadowVault(fail_closed=True)
# 1. Restrict hydration by entity type:
stripe_policy = ToolPolicy(
tool_name="stripe_charge",
allowed_entities={"CREDIT_CARD", "EMAIL"},
fail_closed=True,
fail_action=FailClosedAction.RAISE,
)
# 2. Block hydration completely for untrusted tools:
slack_policy = ToolPolicy(
tool_name="post_slack_alert",
allow_hydration=False, # Receives format-preserving synthetic tokens only
)
@vault_guard(vault=vault, policy=slack_policy)
def post_slack_alert(message: str, card: str, session_id: str = "default"):
# card is guaranteed to remain synthetic (e.g. 4000009995970590)
...Deploy a drop-in proxy for any OpenAI SDK client:
import uvicorn
from shadow_vault import ShadowVault, create_proxy_app
vault = ShadowVault()
app = create_proxy_app(vault=vault, upstream_base_url="https://api.openai.com/v1")
if __name__ == "__main__":
uvicorn.run(app, host="0.0.0.0", port=8000)Point your client to http://localhost:8000/v1:
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:8000/v1",
api_key="your-api-key",
default_headers={"X-Session-ID": "session_42"}
)
# Streaming SSE response with real-time token re-hydration:
stream = client.chat.completions.create(
model="gpt-4o",
messages=[{"role": "user", "content": "Send bill to alice@example.com"}],
stream=True
)
for chunk in stream:
if chunk.choices[0].delta.content:
print(chunk.choices[0].delta.content, end="", flush=True)For workloads requiring Named Entity Recognition (NER) for Person names or custom NLP entities, install the optional Presidio integration without burdening the zero-dependency core:
pip install "shadow-vault[presidio]"Plug the detector directly into your ShadowVault engine:
from shadow_vault import ShadowVault
from shadow_vault.integrations.presidio import PresidioDetector
vault = ShadowVault()
# Plug in Microsoft Presidio / spaCy analyzer:
vault.register_custom_detector(PresidioDetector())
# Dehydrates names with format-preserving synthetic representations:
prompt = vault.dehydrate("Schedule follow-up with Dr. Gregory House.", session_id="s1")
# Prompt: "Schedule follow-up with Anon_Gregory_House."
# Fully reversible at tool boundaries:
hydrated = vault.hydrate(prompt, session_id="s1")
# Restored: "Schedule follow-up with Dr. Gregory House."Measured on a realistic 2,000-token prompt containing 12 distinct PII entities, structured MCP tool-call dictionaries, and 500-chunk SSE streams using benchmarks/run_benchmarks.py:
| Workload / Operation | Input Payload Size | P50 Latency | P95 Latency | P99 Latency | Target Met |
|---|---|---|---|---|---|
| Dehydrate Prompt (12 Entities) | ~2,000 tokens (4.6 KB) | 2.876 ms | 4.043 ms | 4.531 ms | Sub-5ms |
| Hydrate MCP Tool Arguments | 5 Synthetic PII Args | 26.6 μs (0.0266 ms) | 45.9 μs | 73.4 μs | < 0.1 ms |
| StreamHydrator (Per Chunk) | 500 Token Chunks | 50.0 μs (0.0500 ms) | 63.0 μs | 72.3 μs | < 0.1 ms |
| Full 500-Chunk Stream Total | 500 SSE Token Chunks | 25.00 ms | 31.49 ms | 36.16 ms | Real-Time |
| Metric | Measured Footprint | Comparison vs. Heavy NLP (spaCy / Presidio) |
|---|---|---|
| Memory per Active Session | 0.028 KB (28 bytes) | ~12,500x lighter (~500 MB model load vs. 0.04 KB) |
| Peak Heap Overhead (500 Active Sessions) | 28.56 KB | Zero GPU / Torch dependencies required |
| Storage Engine | SQLite In-Memory / WAL | Built-in TTL expiration prunes inactive sessions |
| Test Scenario | Sample Size | False-Positive Rejection Rate | Theoretical Guarantee |
|---|---|---|---|
| Random 16-Digit IDs (Order & Tracking Numbers) | 10,000 | 90.03% | Modulo 10 Luhn algorithm rejects non-card numbers |
| Random 12-Digit IDs (Transaction References) | 10,000 | 89.95% | Verhoeff dihedral group |
| Single-Digit Substitution & Transposition Errors | Exhaustive | 100.0% | 100% mathematical error detection guarantee |
To re-run the benchmark suite locally:
python benchmarks/run_benchmarks.pyShadowVault maintains an explicit, audited threat model and security disclosure policy:
- THREAT_MODEL.md: Trust boundaries, STRIDE attacker vectors, AEAD cryptographic specifications, blind indexing derivations, and explicit non-goals.
- SECURITY.md: Vulnerability disclosure SLA, responsible reporting contacts, and supported release branches.
- Adversarial Security Suite: Automated regression tests verifying cross-session isolation, SQL injection prevention, high-concurrency race condition handling, and bit-flipping ciphertext rejection.
- Reproducible Verification CLI: Run
shadow-vault verifyto independently execute 6 automated security checks covering AEAD roundtrips, blind index entropy, checksum integrity, and fail-closed leak blocking.
- Named Entity Recognition (Person Names): Free-text unstructured names (e.g. "Dr. Gregory House") require the optional Presidio/spaCy NER adapter (
pip install "shadow-vault[presidio]"). The core engine focuses on deterministic identifiers. - Structural Preservation: Format-preserving tokens intentionally maintain length and type (e.g. 16-digit cards). While the value is synthetic, an observer can deduce the field type.
- In-Process Keys: Process memory dumps are out-of-scope for application libraries. Enforce OS-level container isolation and set persistent vault keys via
SHADOW_VAULT_KEY.
pytest -v
# 82 passed in 3.26s (100% pass rate)Contributions are welcome! Read CONTRIBUTING.md to see how you can add a new regional PII detector in under 20 lines of code.
Distributed under the MIT License. See LICENSE for details.