Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 11 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -337,6 +337,17 @@ jobs:
- name: Records name what they cover in scope; the older field stays with old records
run: python scripts/check_scope_migration.py

publish_metadata:
name: no new personal metadata in published files
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
- uses: actions/setup-python@0b93645e9fea7318ecaed2b359559ac225c90a2b # v5.3.0
with:
python-version: "3.13"
- name: A file may not add a local path, address or metadata over the frozen baseline
run: python scripts/publish_metadata_gate.py

file_gate:
name: 300-line gate, burn-down only shrinks
runs-on: ubuntu-latest
Expand Down
3 changes: 3 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,9 @@ lists the changes in each release with the pull request that made them.

## Unreleased

Publish privacy
- `scripts/publish_metadata_gate.py` blocks a publish that adds personal metadata. The 1,370 findings already in the tree are frozen as counts in `project-docs/records/2026-11-04-publish-metadata-baseline.json`; a file that grows its count, a new file with a finding, or any hit from the machine name, the git global email or the untracked deny list fails. The baseline only shrinks. `publish_public.py` runs it on the staging tree before any push and takes `--deny-list`; CI runs it on the checkout. The export no longer copies `scripts/.publish-denylist.txt`, which it would have shipped. See `docs/features/publish-metadata.md`.

Datapipe
- `harness.datapipe` takes one external data source from download to a shareable export with a receipt at each stage: a terms gate that needs a saved snapshot and a verbatim licence quote, `datapipe.record/1` rows with cell-level provenance, cleaning checks that flag and never delete, a revertible change log, and an export that blocks anything not cleared for redistribution. See `docs/DATAPIPE.md`.

Expand Down
47 changes: 47 additions & 0 deletions docs/features/publish-metadata.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,47 @@
# Publish metadata: the check and the gate

A published file can name its author in places nobody reads on screen: EXIF and
PNG text chunks, a PDF `/Author`, a docx creator, a local path in a test fixture,
the build machine's name, or a personal address in a fetch script's User-Agent.
Two scripts handle this before anything ships.

## The check

`scripts/check_publish_metadata.py PATH --mode report|fail|fix` reads every
tracked file and names each finding by category and label, never by value, so a
CI log does not republish it. `fix` strips image, PDF and Office metadata in
place and leaves text alone. Private patterns come from places that never ship:
the machine's own name, the git global `user.email`, and an untracked deny list
of regexes, one per line, from `--deny-list`, the `PUBLISH_DENYLIST` variable, or
the gitignored `scripts/.publish-denylist.txt`.

## The gate

`scripts/publish_metadata_gate.py [ROOT]` blocks a publish that adds metadata.
The public tree carried 1,370 findings in 214 files when the gate was frozen on
2026-11-04, nearly all workspace paths in test fixtures and scripts. Those counts
live in `project-docs/records/2026-11-04-publish-metadata-baseline.json` as path,
category and count, with no matched value. The gate fails when:

- a file's count for a category grows past its frozen count;
- a file outside the baseline has any finding;
- any private pattern matches (machine name, git global email, deny list). The
baseline is written from the generic patterns only, so these always block.

The baseline only shrinks. `--write-baseline` refuses a larger count unless
`--allow-growth` is passed, and a cleaned file drops out of it.

Where it runs:

- `scripts/publish_public.py` runs it over the staging tree after
`publish_lint --strict` and before any push. `--deny-list FILE` passes a deny
list through. The export never copies `scripts/.publish-denylist.txt`.
- CI runs it over the checkout in the `publish_metadata` job, with the generic
patterns only, since the runner holds no deny list.

## Limits

The text patterns are regexes over lines. A path split across two lines, or an
identifier spelled in a form no pattern names, passes. Git author and timezone
findings stay warnings: they describe history, which a noreply address and
`TZ=UTC` fix going forward.
Loading
Loading