Align docs with the code and the production host

[erskingardner]

Problem

The docs no longer describe how Pensieve runs, and some of them would lead an operator or agent to do the wrong thing on production:

• Deploy and rollback. AGENTS.md and the RUNBOOK say to deploy with git pull + just build-release in ~/pensieve and to roll back with git checkout HEAD~1. No service runs ~/pensieve/target/release/. systemd drop-ins point each one at a release build, and the checkout is 29 commits behind master with uncommitted edits that are live: Compose and archive-sync use them.
• Migrations. The docs offer just ch-migrate-all as "run all pending migrations". Nothing records which migrations have run, so it replays all of 001–015 on the live database, including the drop/rebuild migrations and 009's TRUNCATE + backfill. The migrations README says 016 "has not been applied to production" (it was, on 2026-07-27). The 017 rollback has no warning that it is no longer cheap now that 018 has run.
• Schema. docs/clickhouse_self_hosted.sql reads like the production schema. Production events_local has no projections.
• Smaller drift. pensieve-serve is called a placeholder. Three crates are missing from the docs. A documented flag (--import-relays-csv) doesn't exist. pensieve.env is said to configure the ingester (it doesn't). The analytics-build recipe is broken.

Fix

These are doc and comment changes plus repo copies of the production units. I did not change anything on production, and I did not touch the host's checkout or its local edits.

• AGENTS.md and ops/RUNBOOK.md:
  • What each service actually starts.
  • The release pattern: detached worktree → releases/<component>-<sha>/ → drop-in → restart → check /proc/<pid>/exe.
  • Rollback by pointing the drop-in back at the previous release. This includes a warning that deleting 10-release.conf, as its own comment suggests, would start a 2026-07-27 preview build that full-scans events_local.
  • Which units read pensieve.env.
  • Postgres is manual-only.
  • Current crate list.
  • Env-var tables for serve and preview, including PREVIEW_COMPACT_LOOKUPS and PREVIEW_DISABLE_PROJECTIONS.
• ops/systemd/dropins/: copies of the production drop-ins, with a README. They contain no secrets: the blocked giftwrap recipient pubkey is already in crates/pensieve-ingest tests. I also added the pensieve-archive-admission-kuma service, timer and script, which until now existed only on the host. They are copied verbatim.
• Migrations README, base schema header and justfile:
  • ch-migrate-all is for empty databases only. This is stated prominently in all three places and in AGENTS.md.
  • A "Production state" section with the date each projection was dropped.
  • 016/017/018 marked applied, with the 017-rollback warning.
  • The base schema is documented as the fresh-install starting point. Its DDL is unchanged, so Prepare removal of the events_by_kind projection (migration 018) #70's decision to keep events_by_kind for fresh installs still holds.
• Agent-misleading fixes elsewhere:
  • RUNBOOK campaign step: --setenv INVENTORY_ONLY=1 is overridden by the env file's INVENTORY_ONLY=0, so the "read-only" step would have run the full campaign. It now uses /usr/bin/env.
  • ops/analytics.env.example: the Postgres socket address could not work. It now uses 127.0.0.1.
  • env.production.example: added POSTGRES_ANALYTICS_PASSWORD, which is required. Without it, docker compose refuses to start; I verified this locally.
  • README: the flag that doesn't exist is gone.
  • just run-ingest -- --help corrected to just run-ingest --help.
  • analytics-build now passes --bin.
• Descriptive drift:
  • README, env.local.example and the serve README: /stats/engagement returns 503 unless served from Postgres, /stats/events defaults to a 30-day window, and the error codes table is complete.
  • Dated status banners on stale plans and negentropy increment notes.
  • Comment-only fixes in ops/production. I validated compose config, promtool and the XML parse.

Production evidence (read-only, 2026-10-11)

• systemctl cat / systemctl show:
  • pensieve-api → releases/api-7ac5121/pensieve-serve (90-postgres-cutover.conf, PENSIEVE_POSTGRES_API_FAMILIES=none).
  • pensieve-preview → releases/preview-3ab1045/pensieve-preview (10-release.conf).
  • pensieve-ingest → /data/pensieve-build/negentropy-20f0b4b/release/pensieve-ingest (zzzz-bounded-canary.conf, Restart=no). The other three drop-ins are giftwrap-recipient-block.conf, nip66-monitor.conf and override.conf.
• ls:
  • ~/pensieve/releases holds api-{6b21547,8a6bc61,9829378,7ac5121} and preview-3ab1045.
  • ~/releases holds negentropy-* source copies.
  • ~/pensieve/target/release/pensieve-preview is dated 2026-07-27.
• git -C ~/pensieve status: master at 557d241, 29 behind origin/master. Modified: env.production.example, ops/production/compose.yml, ops/scripts/sync-archive.sh. The edited sync-archive.sh is identical to current master, and the other two differ from master only by later additions on master. Detached worktrees exist for giftwrap-hotfix-5ecaca9, pensieve-canary-72da64e, pensieve-preview-3ab1045 and pensieve-slice6-8d004f2.
• /etc/pensieve/pensieve.env (names only): CLICKHOUSE_DB, not CLICKHOUSE_DATABASE (harmless: the binaries default to nostr). POSTGRES_ANALYTICS_PASSWORD and PREVIEW_COMPACT_LOOKUPS are present. No RELAY_DB_PATH.
• Docker: docker ps shows clickhouse, grafana, prometheus and caddy; postgres-analytics is not running.
• ClickHouse 24.8.14.39:
  • SHOW CREATE TABLE nostr.events_local lists no PROJECTION, and system.projection_parts is empty.
  • system.mutations shows DROP PROJECTION events_by_author on 2026-06-13 (manual, no migration), events_by_time on 2026-07-27 and events_by_kind on 2026-10-11 16:54, all is_done=1. /data is at 57%.
• Fresh-install check on a local ClickHouse 24.8: ch-init + ch-migrate-all + prepared 017 yields the same 43 tables and views as production, with identical DDL apart from the projections on events_local. ch-init fails on a bare server until nostr exists; the production compose creates it via CLICKHOUSE_DB.

Findings inventory (line numbers refer to master before this PR)

Misleads an operator or agent

File:line	Doc said	True	Fix
AGENTS.md:264-305, 382-404	Deploy = git pull + just build-release in ~/pensieve; roll back with git checkout HEAD~1 + rebuild	Services run releases/… or /data/pensieve-build/… via drop-ins; the checkout is stale and has live edits	Rewritten around the release-dir pattern; rollback = repoint the drop-in
AGENTS.md:270-276	Binaries in ~/pensieve/target/release/	See evidence above	New architecture table
AGENTS.md:68, 244-247; justfile:247, 253	ch-migrate-all runs "pending" migrations; ch-migrate is "safe to run multiple times"; 009 is the example	No ledger; replays drops, TRUNCATE and backfills	Fresh-DB-only warnings; neutral example
docs/migrations/README.md:391	016 "has not been applied to production"	Applied 2026-07-27	Status lines and index column
docs/migrations/README.md:493	017 rollback with no caveat	After 018 it means 10–30 s scans, and 018 cannot practically be rolled back	Warning
docs/clickhouse_self_hosted.sql:1-7	Reads as the schema	Fresh-install base; production has no projections, events_by_author dropped by hand	Header documents the relationship; DDL unchanged (per #70)
ops/RUNBOOK.md:21-22, 380-383; env.production.example:8-9	pensieve.env configures the native binaries including ingest	The ingest unit doesn't load it	Corrected reader list
ops/RUNBOOK.md:74, 372	install ops/systemd/*.service	Also installs gated units	Install changed units only
ops/RUNBOOK.md:181-188	--setenv=INVENTORY_ONLY=1	EnvironmentFile overrides it, so the full campaign would run	/usr/bin/env INVENTORY_ONLY=1
ops/RUNBOOK.md:8-9, 29	Postgres runs in the compose stack	manual-postgres profile, not running	Documented
ops/RUNBOOK.md:409-418	Build preview-lookups from the checkout	The checkout predates #67	Use the release binary
ops/analytics.env.example:9	Unix socket DATABASE_URL	Postgres is in Docker on 127.0.0.1:5432	Fixed
env.production.example	No POSTGRES_ANALYTICS_PASSWORD	Compose refuses to start without it	Added
README.md:96, 221-235	--import-relays-csv	No such flag	--import-discovered-json
README.md:85	--output	--output-dir	Fixed
env.local.example:8; AGENTS.md:428	just run-ingest -- --help	Double -- breaks clap	Fixed
justfile:72-74	analytics-build	Fails: 34 bins, no default-run	--bin pensieve-analytics
docs/preview_compact_lookups.md	plan/validate/bench usage	Need --allow-full-scan after 018	Note
docs/negentropy_execution_plan.md:50-52	Self-declared authoritative status, pre-#60	#60, #63–#66 merged; production pilot started 2026-10-11	Rows updated
crates/pensieve-serve/README.md	/stats/engagement documented as live; /stats/events "total"	503 unless served from Postgres; 30-day default window	Fixed

Descriptive drift

File:line	Fix
AGENTS.md:41, 91	serve is not a placeholder
AGENTS.md:80-99	Structure omitted analytics, lake, parquet, the newer ingest bins and sync/
AGENTS.md:166	Segments also seal on a 300 s timer
AGENTS.md:182	Test coverage note was wrong
AGENTS.md:203	Dedupe values are a 1-byte status flag
AGENTS.md:373-380	Env table was serve-only and incomplete
AGENTS.md:408-411	Data-dirs table was missing paths
README.md:42-48, 250-254	Status table; .env is loaded only by serve and preview
env.production.example	ROCKSDB_PATH, SEGMENT_SIZE, STORAGE_BOX_HOST/USER and S3_PREFIX are read by nothing; RELAY_DB_PATH was undocumented
justfile comments	precommit, test-preview, CH_CONTAINER, clean-data
ops/RUNBOOK.md	One-time cutover marked historical; compact-lookups status; negentropy section; cleanup debt
docs/*.md	Banners on negentropy increment notes; status notes on migration-plan and bounded_analytics_plan; stability_fixes_plan 1.1 fixed in #22; wrong paths and env names in process_isolation_plan, analytics_slice_a and lake_active_file_catalog
ops/production comments	compose usage needs --env-file; Caddy/.vip/Prometheus/users.xml/config.xml comments contradicted the config
crates/pensieve-serve/src/state.rs:140, 181	Doc comment said the default bind is :3000; it is :8080

Not changed (follow-ups)

• Base schema still creates events_by_author. Production dropped it by hand and no repo migration covers that. Whether fresh installs should keep it is a schema decision, so I only documented it.
• The host checkout (~/pensieve) needs reconciling with master deliberately. Its compose.yml lacks master's archive-alerts.yml Prometheus mount, so those alert rules are probably not loaded in production.
• Possible code issues I noticed but did not change:
  • Prometheus is published on 0.0.0.0:9090 (compose).
  • The ClickHouse grafana user can read every database.
  • cache_policy_for_path in pensieve-serve may never match the nested /api/v1 paths.
  • The clean-data recipe leaves ./segments behind.

🤖 Generated with Claude Code