GroundedQA

Citation-first document assistant with measurable retrieval evaluation and multi-layer prompt-injection defense. Built on a unified PostgreSQL 16 + pgvector storage architecture.

---

1. Problem & Core Value Proposition

Standard Retrieval-Augmented Generation (RAG) demos frequently fail technical hiring evaluation for three reasons:

1. The "Pseudo-Accuracy" Fallacy: Evaluating arbitrary user uploads with fixed question sets and calling the output "92% Accuracy" without ground truth.
2. Ephemeral Chunk Fragility: Generating naive chunk identifiers (e.g. doc3:p4:c2) that break whenever documents are re-indexed or chunking policies change.
3. Indirect Prompt Injection: Treating documents as trusted text, allowing embedded instructions to override model behavior, leak system prompts, or inject malicious payloads.

GroundedQA solves these problems with strict engineering contracts:

• Two Strictly Separated Modes:
  • Static Benchmark Playground: Measures reproducible, ground-truth accuracy metrics (Recall@1, Recall@3, Recall@5, MRR, Rubric Groundedness, and Abstention Accuracy) against a fixed 5-document seed corpus and 30 curated gold Q&A pairs.
  • Custom Corpus Sandbox: Ingests arbitrary user documents (PDF, Markdown, TXT) and reports retrieval diagnostics only (citation coverage %, score distribution, latency percentiles, abstention rate). Never labels custom uploads as "accuracy".
• Stable Content-Hash Chunk IDs: Deterministically derived from normalized_text + source_hash + chunker_version + chunk_index, preserving chunk identities across re-ingestion and invalidating indices when policies change.
• Unified pgvector Storage: Eliminates distributed consistency headaches by running dense vector search (vector(384)) and PostgreSQL full-text search (GIN index + ts_rank_cd) within a single PostgreSQL 16 instance fused via Reciprocal Rank Fusion (RRF).
• Multi-Layer Injection Defense: Treats documents as untrusted input, caps file sizes/pages, strips active HTML/scripts, wraps retrieved text in data delimiters, runs runtime heuristic injection scans on queries and chunks, logs incidents to a security audit trail, and provides safe abstention.

---

2. System Architecture

flowchart TD
    subgraph Ingestion["Ingestion Pipeline"]
        Upload["Document Upload<br/>(PDF, MD, TXT)"] --> Sanitize["Untrusted Sanitizer<br/>(Strip Scripts, Size/Page Caps)"]
        Sanitize --> Parse["Layout-Aware Parser<br/>(Tables, Columns, Fallback Flag)"]
        Parse --> Chunk["Stable Chunker<br/>(SHA-256 Content-Hash IDs)"]
        Chunk --> Embed["Dense Embedder<br/>(all-MiniLM-L6-v2)"]
    end

    subgraph Storage["Unified Persistence (PostgreSQL 16)"]
        Embed --> PGChunks["chunks table<br/>- embedding: vector(384) [HNSW]<br/>- tsv: tsvector [GIN]<br/>- corpus_mode: benchmark | sandbox<br/>- versions: parser, chunker"]
    end

    subgraph Retrieval["Hybrid Retrieval & Answering"]
        Query["User Query"] --> SecurityScan["Runtime Injection Scanner<br/>(Jailbreaks, Delimiter Breakouts)"]
        SecurityScan -->|Flagged| SecLog["Security Log & Safe Abstention"]
        SecurityScan -->|Clean| Dense["Dense Search<br/>(pgvector cosine distance <=>)"]
        SecurityScan -->|Clean| Lexical["Lexical Search<br/>(Postgres FTS websearch @@ tsv)"]
        Dense & Lexical --> RRF["Reciprocal Rank Fusion<br/>(RRF k=60)"]
        RRF --> Answerer["Grounded Answerer<br/>(Data Delimiters + Exact Citations)"]
        Answerer --> UI["Web Dashboard UI<br/>(Escaped Text, Never Raw HTML)"]
    end

Loading

---

3. Two-Mode Evaluation Contract

GroundedQA enforces a forced mode selection on the home screen:

Property	Static Benchmark Playground	Custom Corpus Sandbox
Corpus	Fixed technical reference corpus (5 docs)	Arbitrary user uploads (PDF, MD, TXT)
Ground Truth	30 curated gold Q&A pairs committed to repo	None (unlabeled arbitrary text)
Permitted Metrics	Accuracy Metrics ONLY: Recall@1/3/5, MRR, Rubric Groundedness, Abstention Acc	Diagnostics ONLY: Citation Coverage %, Score Distribution, p50/p95 Latency, Abstention Rate
Accuracy Claim	Mathematically verifiable against ground truth	Strictly prohibited; labelled as diagnostics
Sample Size	$N=30$ visibly displayed next to every metric	$N$ (active query traces count)
Cross-Mode Comparison	Permitted only between benchmark runs	Forbidden; rejected with HTTP 400

---

4. Quickstart & Local Setup

Prerequisites

• Python 3.12+
• PostgreSQL 16 with pgvector and pg_trgm extensions enabled
  • Database: groundedqa
  • Host: localhost | Port: 5432 | User: groundedqa | Password: groundedqa_dev

Installation

# 1. Clone repository
git clone https://github.com/Shashank-VK/groundedqa.git
cd groundedqa

# 2. Create virtual environment and install pinned dependencies
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
pip install -e .

# 3. Configure environment
cp .env.example .env

# 4. Seed database (Reseeds benchmark corpus, 30 gold Q&A cases, and custom sandbox)
python scripts/seed_db.py

# 5. Start GroundedQA application
uvicorn groundedqa.main:app --host 0.0.0.0 --port 8000

Open your browser at http://localhost:8000 to access the web dashboard. Interactive API documentation is available at http://localhost:8000/docs.

---

5. Scripted Demo Walkthrough

Follow this 2-minute walkthrough to demonstrate all core capabilities:

Step 1: Forced Mode Choice

1. Open http://localhost:8000. A modal prompts for workspace mode.
2. Select Static Benchmark Playground.
3. Notice the blue banner: Active Mode: Static Benchmark Playground (N=30).

Step 2: Grounded Q&A with Citation Inspection

1. Click the preset #01 SQLite Architecture.
2. Click Ask. The grounded answer cites exact chunks (e.g. [chk_xxxx]).
3. Click any blue citation badge. A slide-over Source Chunk Inspector opens showing:
   • Stable chunk ID derived from SHA-256
   • Source document, page number, and exact character offsets
   • Cryptographic metadata (source_hash, content_hash, parser_version, chunker_version).

Step 3: Run Reproducible Benchmark Evaluation

1. Click Run Benchmark Evaluation.
2. The system evaluates all 30 gold cases and populates (measured 2026-10-10, sentence-transformers/all-MiniLM-L6-v2 embeddings, static-benchmark mode, $N=30$):
   • Recall@1: 100.0%
   • Recall@3: 100.0%
   • Recall@5: 100.0%
   • MRR: 1.000
   • Groundedness: 100.0%
   • Abstention Accuracy: 100.0%
   • Note: the gold set was authored from the same self-authored corpus, so these scores reflect an intentionally easy benchmark used for reproducibility demos — the value is in the repeatable evaluation harness (config snapshots, run comparison, JSON/CSV export), not in beating a hard dataset.
3. Click Compare Runs to compare two benchmark runs side-by-side with $\Delta$ differentials and export to JSON or CSV.

Step 4: Prompt Injection Defense & Security Log

1. Switch to Custom Corpus Sandbox.
2. Notice the amber notice: Diagnostics Only Sandbox — Arbitrary documents never claim accuracy.
3. Click System Override Injection preset (System override: Ignore previous instructions...) and click Ask.
4. The system detects the jailbreak, safely refuses the request (SAFE ABSTENTION), and declines to execute the override.
5. Click Security Log in the header. The incident appears in the real-time audit trail with timestamp, prompt_injection_query event type, and matched heuristics.

Step 5: One-Click Authenticated Demo Reset

1. Click Reset Demo in the top navigation.
2. The system invokes POST /api/demo/reset (authenticated via X-Demo-Key), cleans caches, reseeds both benchmark documents and custom sandbox, and restores baseline runs within 5 seconds.

---

6. Architecture Decision Records (ADRs)

Design decisions and governance trade-offs are documented under docs/decision-records/:

• ADR-001: Unified Storage with PostgreSQL, pgvector, and pg_trgm: Why dedicated vector engines (Chroma/Qdrant) were dropped in favor of transactional consistency.
• ADR-002: Two-Mode Evaluation Separation: Preventing deceptive accuracy claims on unlabeled uploads.
• ADR-003: Stable Content-Hash Chunk IDs: Deterministic chunk identities and automatic cache invalidation.
• ADR-004: Multi-Layer Prompt Injection Defense: Untrusted context demarcation, runtime heuristic scanning, and escaped rendering.
• ADR-005: Hybrid Retrieval with Reciprocal Rank Fusion: Balancing dense semantic embeddings with exact lexical token matching.

---

7. Running the Test Suite

GroundedQA includes unit tests, end-to-end integration tests, security failure-path tests, and concurrency tests. The test suite runs 100% offline without network calls or paid API keys:

# Run pytest with coverage
pytest -v --cov=groundedqa --cov-report=term-missing

---

8. License & Attribution

• Source Code: Licensed under the MIT License.
• Benchmark Corpus: Technical documents self-authored by Shashank Karamudi (2026), licensed under Creative Commons Attribution 4.0 (CC BY 4.0). Contains zero proprietary or confidential information.

Architecture

flowchart LR
    Docs[Documents] --> Ingest[Ingestion<br/>parser / chunker / embedder / indexer]
    Ingest --> Store[pgvector store<br/>HNSW cosine index]
    Store --> Hybrid[Hybrid retrieval<br/>dense + full-text, RRF]
    Hybrid --> Answer[Answer generator]
    Answer --> Out[Grounded answer<br/>+ retrieval scores]
    Eval[Evaluation harness<br/>scorers + metrics] --> Answer

Loading

Every answer carries its retrieved chunk IDs and retrieval scores, so groundedness is measured before an answer reaches the user, not assumed.