[WIP] feat: PII detection and de-identification (Presidio) — #233

CLAUDE.md

@@ -89,7 +89,8 @@ src/datasure/
 │   └── local.py            # Local file import (csv/xlsx/xls/json/dta/parquet)
 ├── processing/
 │   ├── prep.py             # Data preparation operations (Polars)
-│   └── corrections.py      # Data correction application
+│   ├── corrections.py      # Data correction application
+│   └── pii.py              # PII detection (Presidio) + redaction + flags
 ├── replication/            # Stata/Python replication package export
 │   ├── package_builder.py, script_generators.py, prep_script_generator.py,
 │   ├── py_script_generators.py, py_prep_script_generator.py,
@@ -142,6 +143,18 @@ pattern `output_view_?.py` - never commit them, and never edit them directly
 5. **Reports** (generated output views): charts and tables per check
 6. **Corrections / replication**: apply corrections; export a replication
    package that reproduces the pipeline outside DataSure
+7. **PII gate** (`processing/pii.py`): column-name heuristics + Presidio
+   value scanning flag PII per dataset (`pii_flags_{alias}` in logs db);
+   per-column decisions are mask / hash (HMAC pseudonyms keyed by a
+   per-project salt in `pii_salt`) / code (persisted category codes in
+   `pii_code_map_{alias}`) / drop / keep. All four can become prep steps
+   (hash/code are idempotent so the export gate can safely re-apply), and
+   the export gate in `package_builder.py` enforces the flags regardless
+   (de-identified is the default export mode; the survey key column is
+   never redacted; salt and code maps never leave the local cache in
+   de-identified exports — generated scripts note hash/code steps rather
+   than reproducing them). spaCy models are downloaded at runtime from
+   the UI, never bundled
 
 ### Cache and data locations (`utils/cache_utils.py`)
 

CONTRIBUTING.md

@@ -168,6 +168,18 @@ tests/
 └── views/                      # Tests for Streamlit page scripts
 ```
 
+### PII detection tests and the spaCy model
+
+PII detection (`src/datasure/processing/pii.py`) uses Microsoft Presidio,
+which needs a spaCy language model for value scanning. The English model
+(`en_core_web_sm`) is a **dev dependency** (installed by `uv sync` via a
+direct wheel URL in `[dependency-groups]` — allowed there because groups
+are not published to PyPI), so the Presidio-path tests in
+`tests/processing/test_pii.py` run for real locally and in CI. Tests that
+need the model skip automatically when it is absent. At runtime, end
+users download models from the PII Review section of the Prepare Data
+page instead.
+
 ## Dependency Management
 
 DataSure declares dependencies in `pyproject.toml` and locks exact versions

README.md

@@ -114,6 +114,7 @@ DataSure includes 9 specialized modules for comprehensive survey data quality mo
 - **Local File Support**: CSV, Excel, Stata (.dta), JSON, and Parquet upload with automatic type detection
 - **Multi-Project Organization**: Manage multiple surveys simultaneously
 - **Data Preparation**: Cleaning and transformation workflows
+- **PII Detection & De-identification**: Presidio-based scanning of column names and values; per-column mask, salted-hash pseudonym, category-code, drop, or keep decisions; and a de-identified (default) or with-PII replication export
 
 ### Interactive Dashboards
 

ROADMAP.md

@@ -44,6 +44,7 @@ For feature requests and bug reports, see our [GitHub Issues](https://github.com
 | **Cross-Platform Support** | Windows, macOS, and Linux compatibility | Launched |
 | **Package Distribution** | Available via PyPI (`uv tool install datasure`) | Launched |
 | **Replication Package Export** | Bundle and export full analysis packages | Launched |
+| **PII Detection & De-identification** | Presidio-based PII scanning, redaction prep action, de-identified exports | Launched |
 
 ---
 

docs/USER_GUIDE.md

@@ -255,6 +255,66 @@ Delete unnecessary or problematic data:
 3. Select column(s) or define row filter
 4. Click "Add"
 
+##### Redact Column
+
+Mask all values in a column with a redaction label (e.g. `[PERSON]`,
+`*****`) while keeping the column in place — the primary tool for
+removing PII without losing the dataset's structure:
+
+1. Click "Add data prep step"
+2. Choose "Redact Column(s)"
+3. Select column(s) and set the redaction label
+4. Click "Add"
+
+#### PII Review
+
+Each dataset tab has a **PII Review** section that scans for columns and
+values suspected to contain personally identifiable information (PII):
+
+- **Column-name heuristics** run instantly with no setup: multilingual
+  restricted-word matching (English, Spanish, French, Swahili terms for
+  names, addresses, phones, GPS, ages, SurveyCTO device metadata) plus a
+  sparsity check that flags high-cardinality free-text columns.
+- **Value scanning** uses Microsoft Presidio with a spaCy language model
+  to detect PII *inside* values (person names, phone numbers, emails,
+  locations) on a sample of each text column. Models are small
+  (~15–40 MB) and downloaded from within the app — English is the
+  default; Spanish and French are available from the language selector.
+
+**Workflow**:
+
+1. (Optional) Download the language model to enable value scanning
+2. Click **Scan for PII**
+3. Review flagged columns: what flagged them, the detected entity type,
+   and sample matched values
+4. Set a per-column decision:
+   - **mask** — replace every value with a constant label (`[PERSON]`)
+   - **hash** — replace values with deterministic salted pseudonyms
+     (`PERSON_3fa1b9c2`); the same value always gets the same token, so
+     group-bys, joins, and frequency analysis keep working, while the
+     secret salt (kept in the local cache, never exported) prevents
+     dictionary attacks
+   - **code** — replace values with readable sequential category codes
+     (`VILLAGE_NAME_001`), assigned in random order and persisted so
+     repeated exports stay consistent
+   - **drop** — remove the column entirely
+   - **keep** — leave the column untouched
+5. Click **Apply decisions as prep steps** — masks, hash pseudonyms,
+   category codes, and drops all land in the change log like any other
+   prep step (replayable and removable), and the decisions are also
+   enforced at export time. The project salt and code maps stay in the
+   local cache and never leave it in de-identified exports, so the
+   generated replication scripts note hash/code steps as
+   applied-in-DataSure rather than reproducing them (the exported
+   datasets already carry the tokens)
+
+> **Warning**: De-identification is not anonymization. Even with direct
+> identifiers masked or dropped, respondents may remain identifiable
+> through combinations of the remaining variables (age, location,
+> occupation, household composition). Deterministic pseudonyms (hash and
+> code) also preserve the frequency distribution, so rare categories stay
+> recognizable by their rarity. Review data before sharing.
+
 #### Change Log
 
 All preparation steps are tracked:

pyproject.toml

@@ -45,6 +45,10 @@ dependencies = [
     "keyring>=25.6.0",
     "polars-readstat>=0.5.1",
     "pyyaml>=6.0.1",
+    # PII detection/redaction (spaCy comes transitively; language models are
+    # downloaded at runtime from the app, never bundled)
+    "presidio-analyzer>=2.2.360",
+    "presidio-anonymizer>=2.2.360",
 ]
 
 [project.urls]
@@ -64,6 +68,10 @@ dev = [
     "jupyterlab>=4.5.7",
     "nbconvert>=7.17.0",
     "mistune>=3.2.1",
+    # English spaCy model so tests can exercise the Presidio NER path.
+    # Direct URLs are allowed here because dependency groups are not
+    # published to PyPI; at runtime users download models from the app UI.
+    "en-core-web-sm @ https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-3.8.0/en_core_web_sm-3.8.0-py3-none-any.whl",
 ]
 
 [tool.uv]

src/datasure/models/enums.py

@@ -15,6 +15,7 @@ class PrepActions(Enum):
     transform_column: str = "transform column(s)"
     remove_column: str = "remove column(s)"
     remove_row: str = "remove row(s)"
+    redact_column: str = "redact column(s)"
 
 
 class PrepMethods(Enum):