From d6e6ffa3c793347bc9798468534014c1c19bf2ba Mon Sep 17 00:00:00 2001 From: Jeff Gardner <202880+erskingardner@users.noreply.github.com> Date: Sat, 10 Oct 2026 14:07:45 -0500 Subject: [PATCH 1/6] Add compact ClickHouse lookups for preview profiles, zaps and naddr nstr.to's profile, zap and naddr reads filter events_local by kind, so they depend on the ~1.33 TiB full-row events_by_kind projection; without it they take 20-60+ seconds. Add three small lookup tables (prepared migration 017) fed by insert-triggered views, a bounded backfill tool, and a reversible PREVIEW_COMPACT_LOOKUPS flag that keeps the legacy reads for comparison and rollback. The projection is not touched. - profile_latest / addressable_latest: argMin states keyed by pubkey and (pubkey, kind, d_tag); newest created_at wins, lowest id breaks ties - zap_receipts_by_target: one row per (target, receipt) with migration 009 amount rules and a parser_version for side-by-side corrections - reads collapse duplicates themselves, so results are identical before and after merges, under replays and out-of-order arrival - preview-lookups tool: plan (disk estimate), backfill (projection-enforced, gap preflight, disk/parts guards, resumable checkpoints), status, validate (Rust oracle + legacy classification), bench (projections and query cache off) - integration tests against ClickHouse 24.8 (just ch-test-up && just test-clickhouse) plus a CI job; runbook for staged rollout, rollback and projection-consumer audit Co-Authored-By: Claude Opus 5.5 --- .github/workflows/ci.yml | 32 +- AGENTS.md | 3 + .../src/bin/preview-lookups.rs | 395 ++++++ crates/pensieve-preview/src/config.rs | 35 + crates/pensieve-preview/src/lib.rs | 1 + .../pensieve-preview/src/lookups/backfill.rs | 477 +++++++ crates/pensieve-preview/src/lookups/bench.rs | 641 +++++++++ .../pensieve-preview/src/lookups/compact.rs | 175 +++ crates/pensieve-preview/src/lookups/mod.rs | 378 ++++++ crates/pensieve-preview/src/lookups/ops.rs | 341 +++++ crates/pensieve-preview/src/lookups/oracle.rs | 370 ++++++ crates/pensieve-preview/src/lookups/plan.rs | 668 ++++++++++ crates/pensieve-preview/src/lookups/status.rs | 102 ++ crates/pensieve-preview/src/lookups/target.rs | 246 ++++ .../pensieve-preview/src/lookups/validate.rs | 680 ++++++++++ crates/pensieve-preview/src/query.rs | 97 +- crates/pensieve-preview/src/render/mod.rs | 44 +- crates/pensieve-preview/src/resolve.rs | 25 +- crates/pensieve-preview/src/routes/json.rs | 6 +- crates/pensieve-preview/src/state.rs | 7 + .../pensieve-preview/tests/compact_lookups.rs | 1146 +++++++++++++++++ .../tests/fixtures/base_schema.sql | 101 ++ docs/clickhouse_self_hosted.sql | 5 + docs/migrations/README.md | 31 + .../prepared/017_preview_compact_lookups.sql | 165 +++ .../017_drop_preview_compact_lookups.sql | 32 + docs/preview_compact_lookups.md | 276 ++++ env.local.example | 2 + env.production.example | 5 + justfile | 25 + ops/RUNBOOK.md | 170 +++ 31 files changed, 6611 insertions(+), 70 deletions(-) create mode 100644 crates/pensieve-preview/src/bin/preview-lookups.rs create mode 100644 crates/pensieve-preview/src/lookups/backfill.rs create mode 100644 crates/pensieve-preview/src/lookups/bench.rs create mode 100644 crates/pensieve-preview/src/lookups/compact.rs create mode 100644 crates/pensieve-preview/src/lookups/mod.rs create mode 100644 crates/pensieve-preview/src/lookups/ops.rs create mode 100644 crates/pensieve-preview/src/lookups/oracle.rs create mode 100644 crates/pensieve-preview/src/lookups/plan.rs create mode 100644 crates/pensieve-preview/src/lookups/status.rs create mode 100644 crates/pensieve-preview/src/lookups/target.rs create mode 100644 crates/pensieve-preview/src/lookups/validate.rs create mode 100644 crates/pensieve-preview/tests/compact_lookups.rs create mode 100644 crates/pensieve-preview/tests/fixtures/base_schema.sql create mode 100644 docs/migrations/prepared/017_preview_compact_lookups.sql create mode 100644 docs/migrations/rollback/017_drop_preview_compact_lookups.sql create mode 100644 docs/preview_compact_lookups.md diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 8719b53..e7774ca 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -98,12 +98,42 @@ jobs: # nextest does not run doctests, so cover them separately. - run: cargo test --workspace --doc + clickhouse: + name: ClickHouse integration (24.8) + runs-on: ubuntu-latest + services: + clickhouse: + # Same release line as production. + image: clickhouse/clickhouse-server:24.8 + env: + CLICKHOUSE_SKIP_USER_SETUP: "1" + ports: + - 8123:8123 + options: >- + --ulimit nofile=262144:262144 + --health-cmd "wget -qO- http://localhost:8123/ping || exit 1" + --health-interval 5s + --health-retries 20 + steps: + - uses: actions/checkout@v4 + - name: Install system dependencies + run: sudo apt-get update && sudo apt-get install -y clang libclang-dev protobuf-compiler + - name: Install Rust (stable) + uses: dtolnay/rust-toolchain@stable + - uses: Swatinem/rust-cache@v2 + with: + shared-key: clickhouse + # Same as `just test-clickhouse`. + - run: cargo test -p pensieve-preview --test compact_lookups -- --ignored + env: + PENSIEVE_TEST_CLICKHOUSE_URL: http://localhost:8123 + # Single required status check that aggregates the matrix + other jobs, so branch # protection only needs to require "CI success". ci-success: name: CI success if: always() - needs: [fmt, clippy, test, doctest, ops-test] + needs: [fmt, clippy, test, doctest, ops-test, clickhouse] runs-on: ubuntu-latest steps: - name: Verify all jobs passed diff --git a/AGENTS.md b/AGENTS.md index a1f4d3b..aacee78 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -49,6 +49,7 @@ just test-preview # Run preview tests just test-preview-verbose # Run with output just coverage-preview # HTML coverage report (requires cargo-llvm-cov) just coverage-preview-summary # Coverage summary +just run-preview-lookups --help # Compact lookup plan/backfill/status/validate/bench ``` ### Agent Commit Requirement @@ -67,6 +68,8 @@ just ch-migrate # Run a single migration just ch-migrate-all # Run active top-level docs/migrations/*.sql only just ch-query "" # Run raw SQL query just ch-tables # List tables and views +just ch-test-up # Disposable ClickHouse 24.8 for integration tests +just test-clickhouse # ClickHouse integration tests (needs ch-test-up) ``` ## Project Structure diff --git a/crates/pensieve-preview/src/bin/preview-lookups.rs b/crates/pensieve-preview/src/bin/preview-lookups.rs new file mode 100644 index 0000000..421220c --- /dev/null +++ b/crates/pensieve-preview/src/bin/preview-lookups.rs @@ -0,0 +1,395 @@ +//! Operator tool for the compact preview lookup tables (migration 017). +//! +//! Subcommands: +//! - `plan`: inspect the source, write a shard plan and print a disk estimate +//! - `backfill`: run the plan with resource limits, disk guards and checkpoints +//! - `status`: backfill progress, table sizes and freshness +//! - `validate`: compare compact and legacy reads against the source events +//! - `bench`: measure lookup latency with projections and query cache off +//! +//! See `docs/preview_compact_lookups.md` and `ops/RUNBOOK.md`. + +use std::path::PathBuf; +use std::process::ExitCode; +use std::time::Duration; + +use anyhow::{Context, Result, bail}; +use clap::{Parser, Subcommand}; +use tracing_subscriber::EnvFilter; + +use pensieve_preview::lookups::backfill::{self, BackfillOptions, StopReason}; +use pensieve_preview::lookups::bench::{self, BenchOptions}; +use pensieve_preview::lookups::ops::{ConnectionArgs, GIB, LimitArgs}; +use pensieve_preview::lookups::plan::{self, CheckpointDir, PlanOptions, ShardBy}; +use pensieve_preview::lookups::status; +use pensieve_preview::lookups::target::Target; +use pensieve_preview::lookups::validate::{self, ValidateOptions}; + +/// Manage the compact ClickHouse lookup tables used by pensieve-preview. +#[derive(Parser, Debug)] +#[command(name = "preview-lookups", version)] +struct Args { + #[command(flatten)] + connection: ConnectionArgs, + + #[command(flatten)] + limits: LimitArgs, + + /// Directory for plans and shard checkpoints. + #[arg(long, default_value = "data/preview-lookups", global = true)] + checkpoint_dir: PathBuf, + + #[command(subcommand)] + command: Command, +} + +#[derive(Subcommand, Debug)] +enum Command { + /// Inspect the source rows, write a shard plan and print a disk estimate. + Plan { + /// Targets to plan (repeatable; default all). + #[arg(long = "target", value_enum)] + targets: Vec, + /// Shard dimension. + #[arg(long, value_enum, default_value = "created-at")] + shard_by: ShardBy, + /// Target source rows per created-at shard. + #[arg(long, default_value_t = 2_000_000)] + max_rows_per_shard: u64, + /// Hex prefix length for id-prefix shards (16^n shards). + #[arg(long, default_value_t = 2, value_parser = clap::value_parser!(u8).range(1..=3))] + id_prefix_len: u8, + /// Allow reading the base table without the events_by_kind projection + /// (full-table scans; for rebuilds after the projection is removed). + #[arg(long)] + allow_full_scan: bool, + /// Free-space reserve shown in the estimate, in GiB. + #[arg(long, default_value_t = 100)] + min_free_gib: u64, + /// Replace an existing plan and delete its completed-shard checkpoints. + #[arg(long)] + discard_progress: bool, + }, + /// Run planned shards (resumable; completed shards are skipped). + Backfill { + /// Targets to backfill (repeatable; default all). + #[arg(long = "target", value_enum)] + targets: Vec, + /// Stop after this many minutes (between shards). + #[arg(long)] + max_runtime_mins: Option, + /// Stop after this many shards per target. + #[arg(long)] + max_shards: Option, + /// Seconds to pause between shards. + #[arg(long, default_value_t = 5)] + delay_secs: u64, + /// Minutes that must pass after the view's creation before backfilling. + #[arg(long, default_value_t = 10)] + grace_mins: u64, + /// Free space that must remain on the table's disk, in GiB. + #[arg(long, default_value_t = 100)] + min_free_gib: u64, + /// Pause while the target has more active parts than this. + #[arg(long, default_value_t = 300)] + max_target_parts: u64, + /// Abort a shard reading more than this multiple of its planned rows. + #[arg(long, default_value_t = 4.0)] + max_read_amplification: f64, + /// After all shards, re-insert rows indexed within --gap-margin-mins of + /// the view's creation (belt and braces; reads the target-kind slice). + #[arg(long)] + gap_pass: bool, + /// Gap-closure window half-width in minutes. + #[arg(long, default_value_t = 30)] + gap_margin_mins: u64, + }, + /// Show backfill progress, table sizes and freshness. + Status, + /// Compare compact and legacy reads against the source events. + Validate { + /// Targets to validate (repeatable; default all). + #[arg(long = "target", value_enum)] + targets: Vec, + /// Random keys per target. + #[arg(long, default_value_t = 500)] + sample: usize, + /// Hottest keys per target. + #[arg(long, default_value_t = 50)] + hot: usize, + /// Sampling seed. + #[arg(long, default_value_t = 17)] + seed: u64, + /// Random keys come from hash % 10000 < this. + #[arg(long, default_value_t = 100)] + sample_permyriad: u32, + /// Compare only data indexed at least this many seconds ago. + #[arg(long, default_value_t = 300)] + cutoff_margin_secs: u64, + /// Keys per source query. + #[arg(long, default_value_t = 200)] + batch: usize, + /// Maximum legacy per-key reads for zaps and naddr. + #[arg(long, default_value_t = 100)] + legacy_sample: usize, + /// Allow source reads without the events_by_kind projection. + #[arg(long)] + allow_full_scan: bool, + /// Write the JSON report here. + #[arg(long)] + out: Option, + }, + /// Measure compact (and optionally legacy) lookup latency and storage. + Bench { + /// Measured calls per lookup. + #[arg(long, default_value_t = 30)] + samples: usize, + /// Pubkeys per batch profile lookup. + #[arg(long, default_value_t = 50)] + batch_size: usize, + /// Sampling seed (default: derived from the server clock). + #[arg(long)] + seed: Option, + /// Also measure legacy reads with projections disabled (slow). + #[arg(long)] + include_legacy: bool, + /// Legacy calls per lookup. + #[arg(long, default_value_t = 3)] + legacy_samples: usize, + /// max_execution_time for legacy calls. + #[arg(long, default_value_t = 120)] + legacy_timeout_secs: u64, + /// Drop the server's mark and uncompressed caches before each query. + /// Affects every query on the server. + #[arg(long)] + drop_caches: bool, + /// Latency target in milliseconds. + #[arg(long, default_value_t = 1000)] + target_ms: u64, + /// Allow key sampling without the events_by_kind projection. + #[arg(long)] + allow_full_scan: bool, + /// Write the JSON report here. + #[arg(long)] + out: Option, + }, +} + +fn targets_or_all(targets: Vec) -> Vec { + if targets.is_empty() { + Target::ALL.to_vec() + } else { + let mut seen = Vec::new(); + for t in targets { + if !seen.contains(&t) { + seen.push(t); + } + } + seen + } +} + +fn write_report(path: Option<&PathBuf>, report: &T) -> Result<()> { + if let Some(path) = path { + if let Some(parent) = path.parent() { + std::fs::create_dir_all(parent)?; + } + std::fs::write(path, serde_json::to_vec_pretty(report)?) + .with_context(|| format!("writing {}", path.display()))?; + eprintln!("report written to {}", path.display()); + } + Ok(()) +} + +#[tokio::main] +async fn main() -> ExitCode { + tracing_subscriber::fmt() + .with_env_filter(EnvFilter::try_from_default_env().unwrap_or_else(|_| "info".into())) + .with_writer(std::io::stderr) + .init(); + + match run(Args::parse()).await { + Ok(code) => code, + Err(e) => { + eprintln!("error: {e:#}"); + ExitCode::FAILURE + } + } +} + +async fn run(args: Args) -> Result { + let database = args.connection.clickhouse_database.clone(); + // Tag every tool query so audits of projection use can exclude them; + // backfill and bench override this per query with a more specific tag. + let client = args + .limits + .apply(args.connection.client()) + .with_option("log_comment", "preview-lookups"); + + match args.command { + Command::Plan { + targets, + shard_by, + max_rows_per_shard, + id_prefix_len, + allow_full_scan, + min_free_gib, + discard_progress, + } => { + let options = PlanOptions { + shard_by, + max_rows_per_shard, + id_prefix_len: usize::from(id_prefix_len), + require_projection: !allow_full_scan, + }; + let mut peak_total = 0u64; + for target in targets_or_all(targets) { + let plan = plan::build_plan(&client, &database, target, &options).await?; + CheckpointDir::new(&args.checkpoint_dir, target) + .save_plan(&plan, discard_progress)?; + println!("{}\n", plan::describe_plan(&plan, min_free_gib * GIB)); + peak_total += plan.estimate.peak_bytes; + } + let disk = + pensieve_preview::lookups::ops::table_disk(&client, &database, "events_local") + .await?; + println!( + "combined peak (est): {}; disk {} unreserved: {} of {}", + pensieve_preview::lookups::ops::human_bytes(peak_total), + disk.name, + pensieve_preview::lookups::ops::human_bytes(disk.unreserved_space), + pensieve_preview::lookups::ops::human_bytes(disk.total_space) + ); + Ok(ExitCode::SUCCESS) + } + Command::Backfill { + targets, + max_runtime_mins, + max_shards, + delay_secs, + grace_mins, + min_free_gib, + max_target_parts, + max_read_amplification, + gap_pass, + gap_margin_mins, + } => { + let started = std::time::Instant::now(); + let max_runtime = max_runtime_mins.map(|m| Duration::from_secs(m * 60)); + let mut complete = true; + for target in targets_or_all(targets) { + let checkpoints = CheckpointDir::new(&args.checkpoint_dir, target); + let Some(plan) = checkpoints.load_plan()? else { + bail!( + "no plan for {target}; run `preview-lookups plan --target {target}` first" + ); + }; + if plan.database != database { + bail!( + "plan for {target} targets database {}, not {database}", + plan.database + ); + } + let options = BackfillOptions { + max_runtime: max_runtime.map(|m| m.saturating_sub(started.elapsed())), + max_shards, + delay_between_shards: Duration::from_secs(delay_secs), + grace: Duration::from_secs(grace_mins * 60), + skip_gap_preflight: false, + min_free_bytes: min_free_gib * GIB, + max_target_parts, + parts_wait_timeout: Duration::from_secs(1800), + max_read_amplification, + }; + let report = backfill::run_backfill(&client, &plan, &checkpoints, &options).await?; + println!("{target}: {}", serde_json::to_string(&report)?); + if report.stopped != StopReason::Complete { + complete = false; + break; + } + if gap_pass { + backfill::run_gap_pass( + &client, + &plan, + &checkpoints, + Duration::from_secs(gap_margin_mins * 60), + &options, + ) + .await?; + } + } + if complete { + Ok(ExitCode::SUCCESS) + } else { + eprintln!("backfill stopped before completion; re-run to resume"); + Ok(ExitCode::from(2)) + } + } + Command::Status => { + let statuses = status::collect(&client, &database, &args.checkpoint_dir).await?; + print!("{}", status::render(&statuses)); + Ok(ExitCode::SUCCESS) + } + Command::Validate { + targets, + sample, + hot, + seed, + sample_permyriad, + cutoff_margin_secs, + batch, + legacy_sample, + allow_full_scan, + out, + } => { + let options = ValidateOptions { + targets: targets_or_all(targets), + sample, + hot, + seed, + sample_permyriad, + cutoff_margin: Duration::from_secs(cutoff_margin_secs), + batch, + legacy_sample, + require_projection: !allow_full_scan, + max_examples: 20, + }; + let report = validate::run(&client, &database, &options).await?; + print!("{}", report.render()); + write_report(out.as_ref(), &report)?; + Ok(if report.unexplained() == 0 { + ExitCode::SUCCESS + } else { + ExitCode::FAILURE + }) + } + Command::Bench { + samples, + batch_size, + seed, + include_legacy, + legacy_samples, + legacy_timeout_secs, + drop_caches, + target_ms, + allow_full_scan, + out, + } => { + let options = BenchOptions { + samples, + batch_size, + seed, + include_legacy, + legacy_samples, + legacy_timeout_secs, + drop_caches, + target_ms, + require_projection: !allow_full_scan, + }; + let report = bench::run(&client, &database, &options).await?; + print!("{}", report.render()); + write_report(out.as_ref(), &report)?; + Ok(ExitCode::SUCCESS) + } + } +} diff --git a/crates/pensieve-preview/src/config.rs b/crates/pensieve-preview/src/config.rs index 110913f..1a387fd 100644 --- a/crates/pensieve-preview/src/config.rs +++ b/crates/pensieve-preview/src/config.rs @@ -4,8 +4,11 @@ use std::collections::HashSet; use std::path::Path; use std::sync::Arc; +use anyhow::Context; use nostr::nips::nip19::{FromBech32, Nip19}; +use crate::lookups::CompactLookups; + /// Application configuration. #[derive(Debug, Clone)] pub struct Config { @@ -28,6 +31,10 @@ pub struct Config { /// Set of "VIP" pubkeys (hex) that get longer cache TTLs. /// Loaded from a `.vip` file — one pubkey per line (hex or npub). pub vip_pubkeys: Arc>, + + /// Which lookups read the compact ClickHouse tables (migration 017) + /// instead of `events_local`. Defaults to none (legacy reads). + pub compact_lookups: CompactLookups, } impl Config { @@ -44,6 +51,8 @@ impl Config { /// - `PREVIEW_BASE_URL`: Base URL for links/OG tags (default: "http://localhost:8081") /// - `PREVIEW_SITE_NAME`: Site name (default: "Nostr Preview") /// - `PREVIEW_VIP_FILE`: Path to VIP pubkeys file (default: ".vip") + /// - `PREVIEW_COMPACT_LOOKUPS`: `none` (default), `all`, or a comma list of + /// `profiles`, `zaps`, `addressable`. An unrecognised value is an error. pub fn from_env() -> anyhow::Result { let bind_addr = std::env::var("PREVIEW_BIND_ADDR").unwrap_or_else(|_| "0.0.0.0:8081".to_string()); @@ -65,6 +74,11 @@ impl Config { let vip_file = std::env::var("PREVIEW_VIP_FILE").unwrap_or_else(|_| ".vip".to_string()); let vip_pubkeys = load_vip_file(&vip_file); + let compact_lookups = std::env::var("PREVIEW_COMPACT_LOOKUPS") + .unwrap_or_default() + .parse::() + .context("PREVIEW_COMPACT_LOOKUPS")?; + tracing::info!( bind_addr = %bind_addr, clickhouse_url = %clickhouse_url, @@ -72,6 +86,7 @@ impl Config { site_name = %site_name, vip_count = vip_pubkeys.len(), vip_file = %vip_file, + compact_lookups = %compact_lookups, "preview configuration loaded" ); @@ -82,6 +97,7 @@ impl Config { base_url, site_name, vip_pubkeys: Arc::new(vip_pubkeys), + compact_lookups, }) } } @@ -169,6 +185,7 @@ mod tests { "PREVIEW_BASE_URL", "PREVIEW_SITE_NAME", "PREVIEW_VIP_FILE", + "PREVIEW_COMPACT_LOOKUPS", ]; /// Helper to run config tests with isolated env vars. @@ -213,6 +230,24 @@ mod tests { assert_eq!(config.base_url, "http://localhost:8081"); assert_eq!(config.site_name, "Nostr Preview"); assert!(config.vip_pubkeys.is_empty()); + assert_eq!(config.compact_lookups, CompactLookups::NONE); + }); + } + + #[test] + fn config_compact_lookups_parsed() { + with_env_vars(&[("PREVIEW_COMPACT_LOOKUPS", "profiles,zaps")], || { + let config = Config::from_env().unwrap(); + assert!(config.compact_lookups.profiles); + assert!(config.compact_lookups.zaps); + assert!(!config.compact_lookups.addressable); + }); + } + + #[test] + fn config_compact_lookups_invalid_is_error() { + with_env_vars(&[("PREVIEW_COMPACT_LOOKUPS", "everything")], || { + assert!(Config::from_env().is_err()); }); } diff --git a/crates/pensieve-preview/src/lib.rs b/crates/pensieve-preview/src/lib.rs index afc771a..3f2b1e9 100644 --- a/crates/pensieve-preview/src/lib.rs +++ b/crates/pensieve-preview/src/lib.rs @@ -35,6 +35,7 @@ pub mod config; pub mod error; +pub mod lookups; pub mod query; pub mod render; pub mod resolve; diff --git a/crates/pensieve-preview/src/lookups/backfill.rs b/crates/pensieve-preview/src/lookups/backfill.rs new file mode 100644 index 0000000..90addab --- /dev/null +++ b/crates/pensieve-preview/src/lookups/backfill.rs @@ -0,0 +1,477 @@ +//! Bounded, resumable, idempotent backfill of a compact lookup table. +//! +//! # Why concurrent ingestion cannot leave gaps or double counts +//! +//! - The target's materialized view (created at `T0`) captures every +//! `events_local` insert whose pipeline is built after `T0`, including +//! async-insert flushes. +//! - Before the first shard runs, [`gap_preflight`] requires `now >= T0 + +//! grace`, no INSERT into `events_local` older than `T0` still running, and +//! no async-insert buffer first written before `T0`. So every row whose +//! insert might have missed the view is already committed and visible to +//! every shard, which all start later. +//! - Overlap (a row seen by both the view and a shard, or by a re-run shard) +//! is harmless: profile and naddr states are `argMin` (idempotent), and zap +//! rows are collapsed per receipt id on read. +//! - The optional gap-closure pass re-reads rows with `indexed_at` around +//! `T0` as belt and braces. +//! +//! # Resource bounds +//! +//! Each shard is one `INSERT ... SELECT` with `max_threads`, +//! `max_memory_usage`, external aggregation and `max_execution_time` set by +//! the caller's client. Before each shard the free space on the table's disk +//! must cover the remaining estimate plus a reserve, and the target must not +//! have too many unmerged parts. `max_runtime` and `max_shards` stop between +//! shards; completed shards are checkpointed so a re-run resumes. + +use std::time::{Duration, Instant}; + +use anyhow::{Context, Result, bail}; +use clickhouse::Client; +use serde::Serialize; + +use super::ops::{self, QueryLogEntry}; +use super::plan::{CheckpointDir, Plan, ShardCheckpoint, ShardRange, projection_client}; + +/// How long to wait for a shard's `system.query_log` entry. +const QUERY_LOG_TIMEOUT: Duration = Duration::from_secs(30); + +/// Poll interval while waiting for target merges to catch up. +const PARTS_POLL: Duration = Duration::from_secs(10); + +/// Backfill options. +#[derive(Debug, Clone)] +pub struct BackfillOptions { + /// Stop (resumably) once this much time has elapsed. + pub max_runtime: Option, + /// Stop (resumably) after running this many shards. + pub max_shards: Option, + /// Pause between shards to leave room for live traffic. + pub delay_between_shards: Duration, + /// Minimum time after the view's creation before the first shard. + pub grace: Duration, + /// Skip the gap preflight. Only for tests that control all inserts. + pub skip_gap_preflight: bool, + /// Free space that must remain on the table's disk. + pub min_free_bytes: u64, + /// Wait for merges while the target has more active parts than this. + pub max_target_parts: u64, + /// Give up (resumably) after waiting this long for merges. + pub parts_wait_timeout: Duration, + /// Abort a projection shard that reads more than this multiple of its + /// planned rows (plus one million rows of slack): a sign that some parts + /// lack the projection and were scanned in full. + pub max_read_amplification: f64, +} + +impl Default for BackfillOptions { + fn default() -> Self { + Self { + max_runtime: None, + max_shards: None, + delay_between_shards: Duration::from_secs(5), + grace: Duration::from_secs(600), + skip_gap_preflight: false, + min_free_bytes: super::plan::DEFAULT_MIN_FREE_BYTES, + max_target_parts: 300, + parts_wait_timeout: Duration::from_secs(1800), + max_read_amplification: 4.0, + } + } +} + +/// Why a backfill run stopped. +#[derive(Debug, Clone, PartialEq, Eq, Serialize)] +#[serde(tag = "reason", content = "detail", rename_all = "snake_case")] +pub enum StopReason { + /// Every shard is complete. + Complete, + /// `max_runtime` elapsed. + MaxRuntime, + /// `max_shards` shards ran. + MaxShards, + /// The disk guard refused the next shard. + DiskGuard(String), + /// Target merges did not catch up in time. + PartsBacklog(String), +} + +/// Summary of one backfill run. +#[derive(Debug, Clone, Serialize)] +pub struct BackfillReport { + /// Shards in the plan. + pub shards_total: usize, + /// Shards complete after this run. + pub shards_done: usize, + /// Shards executed by this run. + pub shards_run: usize, + /// Rows written by this run (from `system.query_log`, when available). + pub rows_written: u64, + /// Why the run stopped. + pub stopped: StopReason, +} + +/// Verify that no `events_local` insert can have missed the view. +pub async fn gap_preflight(client: &Client, plan: &Plan, grace: Duration) -> Result<()> { + let view = plan.target.materialized_view(); + let t0 = ops::require_view(client, &plan.database, view).await?; + if t0 != plan.view_created_at { + bail!( + "{}.{view} was (re)created at {t0}, after the plan ({}); re-run `plan`", + plan.database, + plan.view_created_at + ); + } + + let now = ops::server_now(client).await?; + let ready_at = t0.saturating_add(grace.as_secs() as u32); + if now < ready_at { + bail!( + "{view} was created {}s ago; wait until {}s after creation before backfilling", + now.saturating_sub(t0), + grace.as_secs() + ); + } + + let running = ops::inserts_started_before(client, &plan.database, t0).await?; + if running > 0 { + bail!( + "{running} INSERT(s) into events_local started before {view} existed and are still running; retry later" + ); + } + + let queued = ops::async_inserts_queued_before(client, &plan.database, t0).await?; + if queued > 0 { + bail!( + "{queued} async-insert buffer(s) for events_local predate {view}; run \ + SYSTEM FLUSH ASYNC INSERT QUEUE or wait, then retry" + ); + } + Ok(()) +} + +/// Execute the remaining shards of `plan`. +/// +/// `client` should already carry resource limits ([`ops::LimitArgs::apply`]). +pub async fn run_backfill( + client: &Client, + plan: &Plan, + checkpoints: &CheckpointDir, + options: &BackfillOptions, +) -> Result { + if !options.skip_gap_preflight { + gap_preflight(client, plan, options.grace).await?; + } + + let started = Instant::now(); + let done: std::collections::BTreeSet = checkpoints + .completed_shards()? + .into_iter() + .map(|c| c.index) + .collect(); + let total = plan.shards.len(); + let mut shards_done = plan + .shards + .iter() + .filter(|s| done.contains(&s.index)) + .count(); + let mut shards_run = 0usize; + let mut rows_written = 0u64; + let insert_client = projection_client(client, plan.require_projection); + + tracing::info!( + target = %plan.target, + shards_total = total, + shards_done, + "starting backfill" + ); + + for shard in &plan.shards { + if done.contains(&shard.index) { + continue; + } + + if let Some(max) = options.max_runtime + && started.elapsed() >= max + { + return Ok(report( + total, + shards_done, + shards_run, + rows_written, + StopReason::MaxRuntime, + )); + } + if let Some(max) = options.max_shards + && shards_run >= max + { + return Ok(report( + total, + shards_done, + shards_run, + rows_written, + StopReason::MaxShards, + )); + } + + let remaining_rows: u64 = plan + .shards + .iter() + .filter(|s| s.index >= shard.index && !done.contains(&s.index)) + .map(|s| s.estimated_rows) + .sum(); + if let Some(refusal) = disk_guard(client, plan, remaining_rows, options).await? { + return Ok(report( + total, + shards_done, + shards_run, + rows_written, + StopReason::DiskGuard(refusal), + )); + } + if let Some(backlog) = wait_for_merges(client, plan, options).await? { + return Ok(report( + total, + shards_done, + shards_run, + rows_written, + StopReason::PartsBacklog(backlog), + )); + } + + let checkpoint = run_shard( + &insert_client, + client, + plan, + shard.index, + &shard.range, + shard.estimated_rows, + options, + ) + .await?; + let written = checkpoint.query_log.as_ref().map_or(0, |q| q.written_rows); + rows_written += written; + checkpoints.save_shard(&checkpoint)?; + shards_done += 1; + shards_run += 1; + + let elapsed = started.elapsed(); + let eta = if shards_run > 0 && shards_done < total { + let per_shard = elapsed / shards_run as u32; + format!("{:?}", per_shard * (total - shards_done) as u32) + } else { + "-".to_string() + }; + tracing::info!( + target = %plan.target, + shard = shard.index, + progress = format!("{shards_done}/{total}"), + percent = format!("{:.1}", shards_done as f64 * 100.0 / total.max(1) as f64), + read_rows = checkpoint.query_log.as_ref().map_or(0, |q| q.read_rows), + written_rows = written, + shard_ms = checkpoint.elapsed_ms, + elapsed = ?elapsed, + eta = %eta, + "shard complete" + ); + + if shards_done < total && !options.delay_between_shards.is_zero() { + tokio::time::sleep(options.delay_between_shards).await; + } + } + + Ok(report( + total, + shards_done, + shards_run, + rows_written, + StopReason::Complete, + )) +} + +/// Re-insert target-kind rows whose `indexed_at` lies within `margin` of the +/// view's creation time. Idempotent; reads the whole target-kind slice. +pub async fn run_gap_pass( + client: &Client, + plan: &Plan, + checkpoints: &CheckpointDir, + margin: Duration, + options: &BackfillOptions, +) -> Result { + let margin = margin.as_secs() as u32; + let range = ShardRange::IndexedAt { + start: plan.view_created_at.saturating_sub(margin), + end: plan.view_created_at.saturating_add(margin), + }; + let insert_client = projection_client(client, plan.require_projection); + // Read amplification is expected here (indexed_at is not a key column). + let options = BackfillOptions { + max_read_amplification: f64::INFINITY, + ..options.clone() + }; + let checkpoint = run_shard( + &insert_client, + client, + plan, + usize::MAX, + &range, + 0, + &options, + ) + .await?; + checkpoints.save_gap_pass(&checkpoint)?; + tracing::info!( + target = %plan.target, + written_rows = checkpoint.query_log.as_ref().map_or(0, |q| q.written_rows), + "gap-closure pass complete" + ); + Ok(checkpoint) +} + +fn report( + shards_total: usize, + shards_done: usize, + shards_run: usize, + rows_written: u64, + stopped: StopReason, +) -> BackfillReport { + BackfillReport { + shards_total, + shards_done, + shards_run, + rows_written, + stopped, + } +} + +async fn run_shard( + insert_client: &Client, + log_client: &Client, + plan: &Plan, + index: usize, + range: &ShardRange, + estimated_rows: u64, + options: &BackfillOptions, +) -> Result { + let query_id = ops::new_query_id(&format!("preview-lookups-backfill-{}", plan.target)); + let sql = plan.target.backfill_sql(&plan.database, &range.predicate()); + let started = Instant::now(); + + insert_client + .query(&sql) + .with_option("query_id", query_id.as_str()) + .with_option( + "log_comment", + format!("preview-lookups:backfill:{}:{index}", plan.target), + ) + .execute() + .await + .with_context(|| format!("{} shard {index} ({})", plan.target, range.predicate()))?; + + let elapsed_ms = started.elapsed().as_millis() as u64; + let query_log = ops::query_log_entry(log_client, &query_id, QUERY_LOG_TIMEOUT).await?; + if let Some(entry) = &query_log { + check_projection_use(plan, index, estimated_rows, entry, options)?; + } else { + tracing::warn!( + query_id, + "no system.query_log entry; projection use not verified" + ); + } + + Ok(ShardCheckpoint { + index, + range: range.clone(), + query_id, + elapsed_ms, + query_log, + finished_at: ops::server_now(log_client).await?, + }) +} + +fn check_projection_use( + plan: &Plan, + index: usize, + estimated_rows: u64, + entry: &QueryLogEntry, + options: &BackfillOptions, +) -> Result<()> { + if !plan.require_projection { + return Ok(()); + } + let projection = format!("{}.events_local.events_by_kind", plan.database); + if !entry.projections.iter().any(|p| p == &projection) { + bail!( + "{} shard {index} did not use {projection} (used {:?}); stopping before \ + full-table scans. Rows it wrote are harmless.", + plan.target, + entry.projections + ); + } + let limit = estimated_rows as f64 * options.max_read_amplification + 1_000_000.0; + if entry.read_rows as f64 > limit { + bail!( + "{} shard {index} read {} rows for ~{estimated_rows} planned; some parts may \ + lack the projection. Investigate system.projection_parts before continuing.", + plan.target, + entry.read_rows + ); + } + Ok(()) +} + +async fn disk_guard( + client: &Client, + plan: &Plan, + remaining_rows: u64, + options: &BackfillOptions, +) -> Result> { + let disk = ops::table_disk(client, &plan.database, plan.target.table()).await?; + let e = &plan.estimate; + let remaining_fraction = if plan.source.source_rows == 0 { + 1.0 + } else { + (remaining_rows as f64 / plan.source.source_rows as f64).min(1.0) + }; + let needed = (e.peak_bytes as f64 * remaining_fraction) as u64 + options.min_free_bytes; + if disk.unreserved_space < needed { + return Ok(Some(format!( + "disk {} has {} unreserved; remaining backfill needs ~{} plus {} reserve", + disk.name, + ops::human_bytes(disk.unreserved_space), + ops::human_bytes(needed - options.min_free_bytes), + ops::human_bytes(options.min_free_bytes) + ))); + } + Ok(None) +} + +async fn wait_for_merges( + client: &Client, + plan: &Plan, + options: &BackfillOptions, +) -> Result> { + let deadline = Instant::now() + options.parts_wait_timeout; + loop { + let parts = ops::table_parts(client, &plan.database, plan.target.table()).await?; + if parts.parts <= options.max_target_parts { + return Ok(None); + } + if Instant::now() >= deadline { + return Ok(Some(format!( + "{} still has {} active parts (limit {})", + plan.target.table(), + parts.parts, + options.max_target_parts + ))); + } + tracing::info!( + table = plan.target.table(), + parts = parts.parts, + limit = options.max_target_parts, + "waiting for merges" + ); + tokio::time::sleep(PARTS_POLL).await; + } +} diff --git a/crates/pensieve-preview/src/lookups/bench.rs b/crates/pensieve-preview/src/lookups/bench.rs new file mode 100644 index 0000000..13e7c5e --- /dev/null +++ b/crates/pensieve-preview/src/lookups/bench.rs @@ -0,0 +1,641 @@ +//! Latency and storage benchmark for the compact lookups. +//! +//! Every measured query runs with `optimize_use_projections = 0` and +//! `use_query_cache = 0`, uses keys not touched earlier in the run (a fresh +//! seed by default), and is tagged with a unique `log_comment` so server-side +//! duration and rows/bytes read come from `system.query_log`. `drop_caches` +//! additionally drops the mark and uncompressed caches before each query +//! (server-wide; opt-in). The OS page cache cannot be dropped from SQL, so +//! repeated runs may still be warmer than a cold production request. +//! +//! Key sampling reads through the projection and is not measured. + +use std::collections::HashMap; +use std::future::Future; +use std::time::Instant; + +use anyhow::{Context, Result}; +use clickhouse::Client; +use serde::{Deserialize, Serialize}; + +use super::compact; +use super::ops::{self, TableParts}; +use super::plan::projection_client; +use super::target::{D_TAG_EXPR, Target, ZAP_TARGETS_EXPR}; +use crate::error::PreviewError; +use crate::query; + +/// Benchmark options. +#[derive(Debug, Clone)] +pub struct BenchOptions { + /// Measured calls per lookup. + pub samples: usize, + /// Pubkeys per batch profile lookup (the JSON route uses up to 50). + pub batch_size: usize, + /// Sampling seed (`None`: derive from the server clock for fresh keys). + pub seed: Option, + /// Also measure the legacy reads (slow without the projection). + pub include_legacy: bool, + /// Legacy calls per lookup. + pub legacy_samples: usize, + /// `max_execution_time` for legacy calls. + pub legacy_timeout_secs: u64, + /// Drop mark and uncompressed caches before each measured query. + pub drop_caches: bool, + /// Latency target for the pass/fail summary. + pub target_ms: u64, + /// Sample keys through `events_by_kind`. + pub require_projection: bool, +} + +impl Default for BenchOptions { + fn default() -> Self { + Self { + samples: 30, + batch_size: 50, + seed: None, + include_legacy: false, + legacy_samples: 3, + legacy_timeout_secs: 120, + drop_caches: false, + target_ms: 1_000, + require_projection: true, + } + } +} + +/// One measured call. +#[derive(Debug, Clone, Serialize)] +pub struct Measurement { + /// Lookup name. + pub lookup: &'static str, + /// `compact` or `legacy`. + pub source: &'static str, + /// Key (first key for batches). + pub key: String, + /// Client wall time. + pub wall_ms: f64, + /// Error, if the call failed. + pub error: Option, + /// `log_comment` tag. + pub tag: String, + /// Server-side duration from `system.query_log`. + pub server_ms: Option, + /// Rows read. + pub read_rows: Option, + /// Bytes read. + pub read_bytes: Option, +} + +/// Latency summary for one lookup and source. +#[derive(Debug, Clone, Serialize)] +pub struct Summary { + /// Lookup name. + pub lookup: &'static str, + /// `compact` or `legacy`. + pub source: &'static str, + /// Calls. + pub runs: usize, + /// Failed calls. + pub failures: usize, + /// Wall-time percentiles (ms). + pub wall_p50_ms: f64, + /// Wall p95. + pub wall_p95_ms: f64, + /// Wall p99. + pub wall_p99_ms: f64, + /// Wall max. + pub wall_max_ms: f64, + /// Server p50 (ms). + pub server_p50_ms: Option, + /// Server max (ms). + pub server_max_ms: Option, + /// Max rows read. + pub read_rows_max: Option, + /// Max bytes read. + pub read_bytes_max: Option, + /// All calls succeeded and the slowest was under the target. + pub under_target: bool, +} + +/// Storage of one projection on `events_local`. +#[derive(Debug, Clone, Serialize, Deserialize, clickhouse::Row)] +pub struct ProjectionStorage { + /// Projection name. + pub name: String, + /// Rows. + pub rows: u64, + /// Bytes on disk. + pub bytes_on_disk: u64, +} + +/// Storage comparison. +#[derive(Debug, Clone, Serialize)] +pub struct StorageReport { + /// Compact tables. + pub tables: Vec<(String, TableParts)>, + /// `events_local` projections. + pub projections: Vec, +} + +/// Benchmark report. +#[derive(Debug, Clone, Serialize)] +pub struct BenchReport { + /// Run id (prefix of every `log_comment`). + pub run_id: String, + /// Sampling seed. + pub seed: u64, + /// Latency target. + pub target_ms: u64, + /// Whether caches were dropped before each query. + pub drop_caches: bool, + /// Settings applied to every measured query. + pub settings: Vec<(String, String)>, + /// Per-lookup summaries. + pub summaries: Vec, + /// Raw measurements. + pub measurements: Vec, + /// Storage. + pub storage: StorageReport, +} + +const MEASURED_SETTINGS: [(&str, &str); 2] = + [("optimize_use_projections", "0"), ("use_query_cache", "0")]; + +#[derive(Debug, Deserialize, clickhouse::Row)] +struct LogRow { + log_comment: String, + query_duration_ms: u64, + read_rows: u64, + read_bytes: u64, +} + +struct Runner<'a> { + client: &'a Client, + run_id: String, + options: &'a BenchOptions, + measurements: Vec, +} + +impl Runner<'_> { + fn measured_client(&self, tag: &str, legacy: bool) -> Client { + let mut client = self.client.clone(); + for (name, value) in MEASURED_SETTINGS { + client = client.with_option(name, value); + } + client = client.with_option("log_comment", tag); + if legacy { + client = client + .with_option( + "max_execution_time", + self.options.legacy_timeout_secs.to_string(), + ) + .with_option("timeout_overflow_mode", "throw"); + } + client + } + + async fn measure( + &mut self, + lookup: &'static str, + legacy: bool, + key: &str, + call: F, + ) -> Result<()> + where + F: FnOnce(Client) -> Fut, + Fut: Future>, + { + let source = if legacy { "legacy" } else { "compact" }; + let tag = format!( + "{}:{lookup}:{source}:{}", + self.run_id, + self.measurements.len() + ); + if self.options.drop_caches { + for sql in ["SYSTEM DROP MARK CACHE", "SYSTEM DROP UNCOMPRESSED CACHE"] { + self.client + .query(sql) + .execute() + .await + .with_context(|| sql.to_string())?; + } + } + let client = self.measured_client(&tag, legacy); + let started = Instant::now(); + let result = call(client).await; + let wall_ms = started.elapsed().as_secs_f64() * 1000.0; + self.measurements.push(Measurement { + lookup, + source, + key: key.to_string(), + wall_ms, + error: result.err().map(|e| e.to_string()), + tag, + server_ms: None, + read_rows: None, + read_bytes: None, + }); + Ok(()) + } +} + +/// Run the benchmark. `client` should carry resource limits. +pub async fn run(client: &Client, database: &str, options: &BenchOptions) -> Result { + let now = ops::server_now(client).await?; + let seed = options.seed.unwrap_or(u64::from(now)); + let run_id = format!("preview-lookups-bench-{now}-{seed}"); + let sampler = projection_client(client, options.require_projection); + let n = options.samples.max(1); + let batch = options.batch_size.max(1); + + let legacy_needed = if options.include_legacy { + options.legacy_samples + } else { + 0 + }; + let pubkeys: Vec = sample_widening( + &sampler, + n + legacy_needed + n * 10 + n * batch, + |permille, limit| { + format!( + "SELECT pubkey FROM {database}.events_local WHERE kind = 0 \ + AND cityHash64(pubkey, toUInt64({seed})) % 1000 < {permille} \ + GROUP BY pubkey ORDER BY cityHash64(pubkey, toUInt64({seed})) LIMIT {limit}" + ) + }, + ) + .await + .context("sampling pubkeys")?; + let zap_targets: Vec = + sample_widening(&sampler, n + legacy_needed, |permille, limit| { + format!( + "SELECT target FROM (SELECT tags FROM {database}.events_local WHERE kind = 9735) \ + ARRAY JOIN {ZAP_TARGETS_EXPR} AS target \ + WHERE cityHash64(target, toUInt64({seed})) % 1000 < {permille} \ + GROUP BY target ORDER BY cityHash64(target, toUInt64({seed})) LIMIT {limit}" + ) + }) + .await + .context("sampling zap targets")?; + let hot_zap_targets: Vec = sampler + .query(&format!( + "SELECT target FROM (SELECT tags FROM {database}.events_local WHERE kind = 9735) \ + ARRAY JOIN {ZAP_TARGETS_EXPR} AS target \ + GROUP BY target ORDER BY count() DESC, target LIMIT {}", + n.min(10) + )) + .fetch_all::() + .await + .context("sampling hot zap targets")?; + let addressable_kinds = Target::Addressable.kind_predicate(); + let coordinates: Vec<(String, u16, String)> = + sample_widening(&sampler, n + legacy_needed, |permille, limit| { + format!( + "SELECT pubkey, kind, {D_TAG_EXPR} AS d FROM {database}.events_local \ + WHERE {addressable_kinds} \ + AND cityHash64(pubkey, kind, d, toUInt64({seed})) % 1000 < {permille} \ + GROUP BY pubkey, kind, d \ + ORDER BY cityHash64(pubkey, kind, d, toUInt64({seed})) LIMIT {limit}" + ) + }) + .await + .context("sampling naddr coordinates")?; + + let mut runner = Runner { + client, + run_id: run_id.clone(), + options, + measurements: Vec::new(), + }; + + // Disjoint key pools so no measured query reuses keys from another. + let (singles, rest) = pubkeys.split_at(pubkeys.len().min(n)); + let (legacy_pool, rest) = rest.split_at(rest.len().min(legacy_needed)); + let (names_pool, batch_pool) = rest.split_at(rest.len().min(n * 10)); + if batch_pool.len() < n * batch { + tracing::warn!( + available = batch_pool.len(), + wanted = n * batch, + "fewer pubkeys than requested for batch lookups" + ); + } + + for pk in singles { + runner + .measure("profile", false, pk, |c| async move { + compact::fetch_profile(&c, pk).await + }) + .await?; + } + for chunk in batch_pool.chunks(batch) { + runner + .measure("profiles_batch", false, &chunk[0], |c| async move { + compact::fetch_profiles(&c, chunk).await + }) + .await?; + } + for chunk in names_pool.chunks(10) { + runner + .measure("display_names", false, &chunk[0], |c| async move { + compact::fetch_display_names(&c, chunk).await + }) + .await?; + } + for target in zap_targets.iter().take(n) { + runner + .measure("zaps", false, target, |c| async move { + compact::fetch_zap_totals(&c, target).await + }) + .await?; + } + for target in &hot_zap_targets { + runner + .measure("zaps_hot", false, target, |c| async move { + compact::fetch_zap_totals(&c, target).await + }) + .await?; + } + for (pubkey, kind, d) in coordinates.iter().take(n) { + let label = format!("{kind}:{pubkey}:{d}"); + runner + .measure("naddr", false, &label, |c| async move { + compact::fetch_replaceable(&c, *kind, pubkey, d).await + }) + .await?; + } + + if options.include_legacy { + let m = options.legacy_samples; + for pk in legacy_pool.iter().take(m) { + runner + .measure("profile", true, pk, |c| async move { + query::fetch_profile(&c, pk).await + }) + .await?; + } + for target in zap_targets.iter().skip(n).take(m) { + runner + .measure("zaps", true, target, |c| async move { + query::fetch_zap_totals(&c, target).await + }) + .await?; + } + for (pubkey, kind, d) in coordinates.iter().skip(n).take(m) { + let label = format!("{kind}:{pubkey}:{d}"); + runner + .measure("naddr", true, &label, |c| async move { + query::fetch_replaceable(&c, *kind, pubkey, d).await + }) + .await?; + } + } + + let mut measurements = runner.measurements; + attach_query_log(client, &run_id, &mut measurements).await?; + let summaries = summarize(&measurements, options.target_ms); + let storage = storage(client, database).await?; + + Ok(BenchReport { + run_id, + seed, + target_ms: options.target_ms, + drop_caches: options.drop_caches, + settings: MEASURED_SETTINGS + .iter() + .map(|(k, v)| (k.to_string(), v.to_string())) + .collect(), + summaries, + measurements, + storage, + }) +} + +/// Sample up to `limit` keys, widening the hashed fraction (per mille) until +/// enough keys are found or every key is eligible. +async fn sample_widening( + client: &Client, + limit: usize, + sql: impl Fn(u32, usize) -> String, +) -> Result> +where + T: clickhouse::Row + serde::de::DeserializeOwned, +{ + let mut keys = Vec::new(); + for permille in [5, 50, 500, 1000] { + keys = client.query(&sql(permille, limit)).fetch_all::().await?; + if keys.len() >= limit { + break; + } + } + Ok(keys) +} + +async fn attach_query_log( + client: &Client, + run_id: &str, + measurements: &mut [Measurement], +) -> Result<()> { + let expected = measurements.len(); + let mut rows: HashMap = HashMap::new(); + for _ in 0..30 { + ops::flush_logs(client).await; + rows = client + .query( + "SELECT log_comment, query_duration_ms, read_rows, read_bytes \ + FROM system.query_log \ + WHERE event_date >= yesterday() AND type != 'QueryStart' \ + AND startsWith(log_comment, ?)", + ) + .bind(format!("{run_id}:")) + .fetch_all::() + .await + .context("reading system.query_log")? + .into_iter() + .map(|r| (r.log_comment.clone(), r)) + .collect(); + if rows.len() >= expected { + break; + } + tokio::time::sleep(std::time::Duration::from_secs(1)).await; + } + for m in measurements.iter_mut() { + if let Some(row) = rows.get(&m.tag) { + m.server_ms = Some(row.query_duration_ms); + m.read_rows = Some(row.read_rows); + m.read_bytes = Some(row.read_bytes); + } + } + Ok(()) +} + +/// Nearest-rank percentile of sorted values. +fn percentile(sorted: &[T], p: f64) -> Option { + if sorted.is_empty() { + return None; + } + let rank = ((p / 100.0) * sorted.len() as f64).ceil() as usize; + Some(sorted[rank.clamp(1, sorted.len()) - 1]) +} + +fn summarize(measurements: &[Measurement], target_ms: u64) -> Vec { + let mut groups: Vec<((&'static str, &'static str), Vec<&Measurement>)> = Vec::new(); + for m in measurements { + match groups.iter_mut().find(|(k, _)| *k == (m.lookup, m.source)) { + Some((_, v)) => v.push(m), + None => groups.push(((m.lookup, m.source), vec![m])), + } + } + groups + .into_iter() + .map(|((lookup, source), ms)| { + let mut wall: Vec = ms.iter().map(|m| m.wall_ms).collect(); + wall.sort_by(f64::total_cmp); + let mut server: Vec = ms.iter().filter_map(|m| m.server_ms).collect(); + server.sort_unstable(); + let failures = ms.iter().filter(|m| m.error.is_some()).count(); + let wall_max = wall.last().copied().unwrap_or(0.0); + Summary { + lookup, + source, + runs: ms.len(), + failures, + wall_p50_ms: percentile(&wall, 50.0).unwrap_or(0.0), + wall_p95_ms: percentile(&wall, 95.0).unwrap_or(0.0), + wall_p99_ms: percentile(&wall, 99.0).unwrap_or(0.0), + wall_max_ms: wall_max, + server_p50_ms: percentile(&server, 50.0), + server_max_ms: server.last().copied(), + read_rows_max: ms.iter().filter_map(|m| m.read_rows).max(), + read_bytes_max: ms.iter().filter_map(|m| m.read_bytes).max(), + under_target: failures == 0 && wall_max < target_ms as f64, + } + }) + .collect() +} + +async fn storage(client: &Client, database: &str) -> Result { + let mut tables = Vec::new(); + for target in Target::ALL { + let parts = ops::table_parts(client, database, target.table()).await?; + tables.push((target.table().to_string(), parts)); + } + tables.push(( + "events_local".to_string(), + ops::table_parts(client, database, "events_local").await?, + )); + let projections = client + .query( + "SELECT name, sum(rows) AS rows, sum(bytes_on_disk) AS bytes_on_disk \ + FROM system.projection_parts \ + WHERE database = ? AND table = 'events_local' AND active \ + GROUP BY name ORDER BY name", + ) + .bind(database) + .fetch_all::() + .await + .context("reading system.projection_parts")?; + Ok(StorageReport { + tables, + projections, + }) +} + +impl BenchReport { + /// Text summary. + pub fn render(&self) -> String { + let mut out = format!( + "run {} (seed {}), settings {:?}, drop caches: {}, target {} ms\n\n", + self.run_id, self.seed, self.settings, self.drop_caches, self.target_ms + ); + out.push_str( + "lookup source runs fail p50 ms p95 ms p99 ms max ms srv p50 srv max max rows read under target\n", + ); + for s in &self.summaries { + out.push_str(&format!( + "{:<15} {:<8} {:>4} {:>4} {:>8.1} {:>8.1} {:>8.1} {:>8.1} {:>8} {:>8} {:>14} {}\n", + s.lookup, + s.source, + s.runs, + s.failures, + s.wall_p50_ms, + s.wall_p95_ms, + s.wall_p99_ms, + s.wall_max_ms, + s.server_p50_ms + .map_or_else(|| "-".into(), |v| v.to_string()), + s.server_max_ms + .map_or_else(|| "-".into(), |v| v.to_string()), + s.read_rows_max + .map_or_else(|| "-".into(), |v| v.to_string()), + if s.under_target { "yes" } else { "NO" }, + )); + } + out.push_str("\nstorage:\n"); + for (table, parts) in &self.storage.tables { + out.push_str(&format!( + " {:<24} {:>14} rows {:>12} on disk ({} parts)\n", + table, + parts.rows, + ops::human_bytes(parts.bytes_on_disk), + parts.parts + )); + } + for p in &self.storage.projections { + out.push_str(&format!( + " events_local.{:<11} {:>14} rows {:>12} on disk (projection)\n", + p.name, + p.rows, + ops::human_bytes(p.bytes_on_disk) + )); + } + out + } +} + +#[cfg(test)] +mod tests { + use super::*; + + fn m(lookup: &'static str, wall_ms: f64, error: bool) -> Measurement { + Measurement { + lookup, + source: "compact", + key: String::new(), + wall_ms, + error: error.then(|| "boom".to_string()), + tag: String::new(), + server_ms: Some(wall_ms as u64), + read_rows: Some(10), + read_bytes: Some(100), + } + } + + #[test] + fn percentile_nearest_rank() { + let v: Vec = (1..=100).collect(); + assert_eq!(percentile(&v, 50.0), Some(50)); + assert_eq!(percentile(&v, 95.0), Some(95)); + assert_eq!(percentile(&v, 100.0), Some(100)); + assert_eq!(percentile(&[7u64], 99.0), Some(7)); + assert_eq!(percentile::(&[], 50.0), None); + } + + #[test] + fn summary_flags_failures_and_slow_calls() { + let ms = vec![ + m("a", 10.0, false), + m("a", 20.0, false), + m("b", 5.0, true), + m("c", 1500.0, false), + ]; + let s = summarize(&ms, 1000); + let get = |l: &str| s.iter().find(|x| x.lookup == l).unwrap(); + assert!(get("a").under_target); + assert_eq!(get("a").runs, 2); + assert_eq!(get("a").wall_max_ms, 20.0); + assert!(!get("b").under_target); + assert_eq!(get("b").failures, 1); + assert!(!get("c").under_target); + } +} diff --git a/crates/pensieve-preview/src/lookups/compact.rs b/crates/pensieve-preview/src/lookups/compact.rs new file mode 100644 index 0000000..270dbc8 --- /dev/null +++ b/crates/pensieve-preview/src/lookups/compact.rs @@ -0,0 +1,175 @@ +//! Reads against the compact lookup tables created by +//! `docs/migrations/prepared/017_preview_compact_lookups.sql`. +//! +//! Every query collapses duplicate rows or aggregate states itself +//! (`argMinMerge` / `GROUP BY receipt_id`), so results are the same before and +//! after background merges. None of these reads touch the `events_by_kind` +//! projection: lookups are primary-key range reads on small tables, and naddr +//! rows are then fetched from `events_local` by event id. + +use std::collections::HashMap; + +use clickhouse::Client; +use serde::Deserialize; + +use crate::error::PreviewError; +use crate::query::{EventRow, ProfileRow, ZapTotals, display_names_from_profiles}; + +/// Whether `addressable_latest` covers a kind (replaceable 10000-19999 and +/// addressable 30000-39999). Other naddr kinds use the legacy query. +pub fn addressable_kind_covered(kind: u16) -> bool { + (10_000..20_000).contains(&kind) || (30_000..40_000).contains(&kind) +} + +/// Fetch the latest profile (kind 0) for a pubkey from `profile_latest`. +pub async fn fetch_profile( + client: &Client, + pubkey: &str, +) -> Result, PreviewError> { + let row = client + .query( + "SELECT pubkey, tupleElement(winner, 1) AS event_id, \ + tupleElement(winner, 2) AS content \ + FROM ( \ + SELECT pubkey, argMinMerge(latest) AS winner \ + FROM profile_latest \ + WHERE pubkey = ? \ + GROUP BY pubkey \ + )", + ) + .bind(pubkey) + .fetch_optional::() + .await?; + + Ok(row) +} + +/// Batch-fetch latest profiles for a set of pubkeys from `profile_latest`. +/// +/// Pubkeys without a profile are omitted. +pub async fn fetch_profiles( + client: &Client, + pubkeys: &[String], +) -> Result, PreviewError> { + if pubkeys.is_empty() { + return Ok(HashMap::new()); + } + + let placeholders: Vec<&str> = pubkeys.iter().map(|_| "?").collect(); + let query_str = format!( + "SELECT pubkey, tupleElement(winner, 1) AS event_id, \ + tupleElement(winner, 2) AS content \ + FROM ( \ + SELECT pubkey, argMinMerge(latest) AS winner \ + FROM profile_latest \ + WHERE pubkey IN ({}) \ + GROUP BY pubkey \ + )", + placeholders.join(", ") + ); + + let mut query = client.query(&query_str); + for pk in pubkeys { + query = query.bind(pk); + } + + let rows = query.fetch_all::().await?; + Ok(rows.into_iter().map(|r| (r.pubkey.clone(), r)).collect()) +} + +/// Batch-fetch display names for a set of pubkeys from `profile_latest`. +pub async fn fetch_display_names( + client: &Client, + pubkeys: &[String], +) -> Result, PreviewError> { + let profiles = fetch_profiles(client, pubkeys).await?; + Ok(display_names_from_profiles( + profiles.into_values().map(|row| (row.pubkey, row.content)), + )) +} + +/// Fetch the latest replaceable/addressable event for a coordinate via the +/// `addressable_latest` pointer and an `events_local` primary-key read. +/// +/// Callers must check [`addressable_kind_covered`] first. +pub async fn fetch_replaceable( + client: &Client, + kind: u16, + pubkey: &str, + d_tag: &str, +) -> Result, PreviewError> { + let row = client + .query( + "SELECT id, pubkey, created_at, kind, content, tags, \ + d_tag, title, thumbnail, video_url \ + FROM events_local \ + WHERE id IN ( \ + SELECT argMinMerge(latest) \ + FROM addressable_latest \ + WHERE pubkey = ? AND kind = ? AND d_tag = ? \ + ) \ + LIMIT 1", + ) + .bind(pubkey) + .bind(kind) + .bind(d_tag) + .fetch_optional::() + .await?; + + Ok(row) +} + +/// Fetch zap totals for a target event from `zap_receipts_by_target`. +/// +/// Rows are collapsed per receipt id before summing, so replayed receipts are +/// counted once even before merges, and the highest `parser_version` wins. +/// Only receipts the parser accepted are counted (the migration 009 rule). +pub async fn fetch_zap_totals(client: &Client, event_id: &str) -> Result { + #[derive(Debug, Deserialize, clickhouse::Row)] + struct Row { + total_msats: u64, + zap_count: u64, + } + + let row = client + .query( + "SELECT sumIf(receipt_msats, receipt_parsed = 1) AS total_msats, \ + countIf(receipt_parsed = 1) AS zap_count \ + FROM ( \ + SELECT receipt_id, \ + argMax(amount_msats, parser_version) AS receipt_msats, \ + argMax(amount_parsed, parser_version) AS receipt_parsed \ + FROM zap_receipts_by_target \ + WHERE target_event_id = ? \ + GROUP BY receipt_id \ + )", + ) + .bind(event_id) + .fetch_optional::() + .await?; + + Ok(row.map_or_else(ZapTotals::default, |r| ZapTotals { + total_msats: r.total_msats, + zap_count: r.zap_count, + })) +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn addressable_coverage_bounds() { + assert!(!addressable_kind_covered(0)); + assert!(!addressable_kind_covered(9_999)); + assert!(addressable_kind_covered(10_000)); + assert!(addressable_kind_covered(10_002)); + assert!(addressable_kind_covered(19_999)); + assert!(!addressable_kind_covered(20_000)); + assert!(!addressable_kind_covered(29_999)); + assert!(addressable_kind_covered(30_000)); + assert!(addressable_kind_covered(30_023)); + assert!(addressable_kind_covered(39_999)); + assert!(!addressable_kind_covered(40_000)); + } +} diff --git a/crates/pensieve-preview/src/lookups/mod.rs b/crates/pensieve-preview/src/lookups/mod.rs new file mode 100644 index 0000000..5861ab4 --- /dev/null +++ b/crates/pensieve-preview/src/lookups/mod.rs @@ -0,0 +1,378 @@ +//! Compact ClickHouse lookups for preview reads. +//! +//! Profiles, zap totals and naddr coordinates can be read either from +//! `events_local` (legacy, projection-dependent) or from the compact lookup +//! tables created by `docs/migrations/prepared/017_preview_compact_lookups.sql`. +//! [`Lookups`] picks the source per lookup from [`CompactLookups`], which is +//! configured with `PREVIEW_COMPACT_LOOKUPS`, so each switch is reversible by a +//! restart while the legacy reads stay available. +//! +//! The submodules other than [`compact`] implement the `preview-lookups` +//! operator tool: bounded backfill, validation against the legacy reads, and a +//! latency/storage benchmark. See `docs/preview_compact_lookups.md`. + +pub mod backfill; +pub mod bench; +pub mod compact; +pub mod ops; +pub mod oracle; +pub mod plan; +pub mod status; +pub mod target; +pub mod validate; + +use std::collections::HashMap; +use std::fmt; +use std::str::FromStr; +use std::time::Instant; + +use clickhouse::Client; + +use crate::error::PreviewError; +use crate::query::{self, EngagementCounts, EventRow, ProfileRow}; + +/// Server-side execution cap for compact lookups, in seconds. +const COMPACT_MAX_EXECUTION_SECS: &str = "5"; + +/// Lookups slower than this are logged at WARN. +const SLOW_LOOKUP_MS: u128 = 1_000; + +/// Which preview lookups read the compact tables instead of `events_local`. +#[derive(Debug, Clone, Copy, Default, PartialEq, Eq)] +pub struct CompactLookups { + /// Kind 0 profiles and display names from `profile_latest`. + pub profiles: bool, + /// Zap totals from `zap_receipts_by_target`. + pub zaps: bool, + /// naddr coordinates from `addressable_latest`. + pub addressable: bool, +} + +impl CompactLookups { + /// Every lookup uses the compact tables. + pub const ALL: Self = Self { + profiles: true, + zaps: true, + addressable: true, + }; + + /// Every lookup uses the legacy `events_local` reads. + pub const NONE: Self = Self { + profiles: false, + zaps: false, + addressable: false, + }; +} + +/// Error returned for an unrecognised `PREVIEW_COMPACT_LOOKUPS` value. +#[derive(Debug, thiserror::Error)] +#[error( + "invalid compact lookup setting {0:?}: expected `none`, `all`, or a comma list of `profiles`, `zaps`, `addressable`" +)] +pub struct ParseCompactLookupsError(String); + +impl FromStr for CompactLookups { + type Err = ParseCompactLookupsError; + + fn from_str(value: &str) -> Result { + let tokens: Vec = value + .split(',') + .map(|t| t.trim().to_ascii_lowercase()) + .filter(|t| !t.is_empty()) + .collect(); + + match tokens.as_slice() { + [] => return Ok(Self::NONE), + [only] if only == "none" => return Ok(Self::NONE), + [only] if only == "all" => return Ok(Self::ALL), + _ => {} + } + + let mut lookups = Self::NONE; + for token in &tokens { + match token.as_str() { + "profiles" => lookups.profiles = true, + "zaps" => lookups.zaps = true, + "addressable" => lookups.addressable = true, + _ => return Err(ParseCompactLookupsError(value.to_string())), + } + } + Ok(lookups) + } +} + +impl fmt::Display for CompactLookups { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + if *self == Self::NONE { + return f.write_str("none"); + } + if *self == Self::ALL { + return f.write_str("all"); + } + let names: Vec<&str> = [ + (self.profiles, "profiles"), + (self.zaps, "zaps"), + (self.addressable, "addressable"), + ] + .into_iter() + .filter_map(|(on, name)| on.then_some(name)) + .collect(); + f.write_str(&names.join(",")) + } +} + +/// Preview read layer that routes each lookup to the legacy or compact source. +/// +/// Every query carries `log_comment = 'pensieve-preview::'` +/// so `system.query_log` can compare latency per lookup and source during +/// rollout. +#[derive(Clone)] +pub struct Lookups { + client: Client, + compact: CompactLookups, +} + +impl Lookups { + /// Create a read layer over `client` with the given compact selection. + pub fn new(client: Client, compact: CompactLookups) -> Self { + Self { client, compact } + } + + /// The configured compact selection. + pub fn compact(&self) -> CompactLookups { + self.compact + } + + fn client_for(&self, lookup: &str, compact: bool) -> Client { + let source = if compact { "compact" } else { "legacy" }; + let client = self + .client + .clone() + .with_option("log_comment", format!("pensieve-preview:{lookup}:{source}")); + if compact { + client + .with_option("max_execution_time", COMPACT_MAX_EXECUTION_SECS) + .with_option("timeout_overflow_mode", "throw") + } else { + client + } + } + + /// Fetch a single event by id (primary-key read; no compact variant). + pub async fn fetch_event_by_id( + &self, + event_id: &str, + ) -> Result, PreviewError> { + let client = self.client_for("event", false); + timed("event", false, query::fetch_event_by_id(&client, event_id)).await + } + + /// Fetch the latest kind 0 profile for a pubkey. + pub async fn fetch_profile(&self, pubkey: &str) -> Result, PreviewError> { + let compact = self.compact.profiles; + let client = self.client_for("profile", compact); + if compact { + timed("profile", true, compact::fetch_profile(&client, pubkey)).await + } else { + timed("profile", false, query::fetch_profile(&client, pubkey)).await + } + } + + /// Batch-fetch latest kind 0 profiles. + pub async fn fetch_profiles( + &self, + pubkeys: &[String], + ) -> Result, PreviewError> { + let compact = self.compact.profiles; + let client = self.client_for("profiles", compact); + if compact { + timed("profiles", true, compact::fetch_profiles(&client, pubkeys)).await + } else { + timed("profiles", false, query::fetch_profiles(&client, pubkeys)).await + } + } + + /// Batch-fetch display names. + pub async fn fetch_display_names( + &self, + pubkeys: &[String], + ) -> Result, PreviewError> { + let compact = self.compact.profiles; + let client = self.client_for("display_names", compact); + if compact { + timed( + "display_names", + true, + compact::fetch_display_names(&client, pubkeys), + ) + .await + } else { + timed( + "display_names", + false, + query::fetch_display_names(&client, pubkeys), + ) + .await + } + } + + /// Fetch the latest event for an naddr coordinate. + /// + /// Kinds outside the `addressable_latest` coverage always use the legacy + /// query. + pub async fn fetch_replaceable( + &self, + kind: u16, + pubkey: &str, + d_tag: &str, + ) -> Result, PreviewError> { + let compact = self.compact.addressable && compact::addressable_kind_covered(kind); + let client = self.client_for("replaceable", compact); + if compact { + timed( + "replaceable", + true, + compact::fetch_replaceable(&client, kind, pubkey, d_tag), + ) + .await + } else { + timed( + "replaceable", + false, + query::fetch_replaceable(&client, kind, pubkey, d_tag), + ) + .await + } + } + + /// Fetch engagement counts; zap totals come from the configured source. + pub async fn fetch_engagement(&self, event_id: &str) -> Result { + let compact = self.compact.zaps; + let counts_client = self.client_for("engagement", false); + let zap_client = self.client_for("zaps", compact); + let zaps = async { + if compact { + timed( + "zaps", + true, + compact::fetch_zap_totals(&zap_client, event_id), + ) + .await + } else { + timed( + "zaps", + false, + query::fetch_zap_totals(&zap_client, event_id), + ) + .await + } + }; + query::fetch_engagement_with_zaps(&counts_client, event_id, zaps).await + } +} + +async fn timed( + lookup: &'static str, + compact: bool, + work: impl std::future::Future>, +) -> Result { + let started = Instant::now(); + let result = work.await; + let elapsed_ms = started.elapsed().as_millis(); + let source = if compact { "compact" } else { "legacy" }; + if elapsed_ms >= SLOW_LOOKUP_MS { + tracing::warn!( + lookup, + source, + elapsed_ms, + ok = result.is_ok(), + "slow preview lookup" + ); + } else { + tracing::debug!( + lookup, + source, + elapsed_ms, + ok = result.is_ok(), + "preview lookup" + ); + } + result +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn parse_none_and_empty() { + assert_eq!("".parse::().unwrap(), CompactLookups::NONE); + assert_eq!( + " , ".parse::().unwrap(), + CompactLookups::NONE + ); + assert_eq!( + "none".parse::().unwrap(), + CompactLookups::NONE + ); + assert_eq!( + "NONE".parse::().unwrap(), + CompactLookups::NONE + ); + } + + #[test] + fn parse_all() { + assert_eq!( + "all".parse::().unwrap(), + CompactLookups::ALL + ); + assert_eq!( + "profiles,zaps,addressable" + .parse::() + .unwrap(), + CompactLookups::ALL + ); + } + + #[test] + fn parse_subsets() { + let profiles = "profiles".parse::().unwrap(); + assert!(profiles.profiles && !profiles.zaps && !profiles.addressable); + + let two = " Zaps , addressable ".parse::().unwrap(); + assert!(!two.profiles && two.zaps && two.addressable); + + let repeated = "profiles,profiles".parse::().unwrap(); + assert!(repeated.profiles && !repeated.zaps); + } + + #[test] + fn parse_rejects_unknown_and_mixed_keywords() { + assert!("profile".parse::().is_err()); + assert!("none,profiles".parse::().is_err()); + assert!("all,zaps".parse::().is_err()); + assert!("yes".parse::().is_err()); + } + + #[test] + fn display_round_trips() { + for value in [ + "none", + "all", + "profiles", + "zaps", + "addressable", + "profiles,zaps", + ] { + let parsed = value.parse::().unwrap(); + assert_eq!(parsed.to_string(), value); + assert_eq!( + parsed.to_string().parse::().unwrap(), + parsed + ); + } + let zaps_addr = "addressable,zaps".parse::().unwrap(); + assert_eq!(zaps_addr.to_string(), "zaps,addressable"); + } +} diff --git a/crates/pensieve-preview/src/lookups/ops.rs b/crates/pensieve-preview/src/lookups/ops.rs new file mode 100644 index 0000000..3e662c1 --- /dev/null +++ b/crates/pensieve-preview/src/lookups/ops.rs @@ -0,0 +1,341 @@ +//! Shared ClickHouse plumbing for the `preview-lookups` operator tool: +//! connection and resource-limit arguments, `system.*` probes and query-log +//! lookups. + +use std::sync::atomic::{AtomicU64, Ordering}; +use std::time::{Duration, SystemTime, UNIX_EPOCH}; + +use anyhow::{Context, Result, bail}; +use clickhouse::Client; +use serde::{Deserialize, Serialize}; + +/// Bytes per GiB. +pub const GIB: u64 = 1024 * 1024 * 1024; + +/// ClickHouse connection arguments. +#[derive(Debug, Clone, clap::Args)] +pub struct ConnectionArgs { + /// ClickHouse HTTP URL. + #[arg(long, env = "CLICKHOUSE_URL", default_value = "http://localhost:8123")] + pub clickhouse_url: String, + + /// ClickHouse database holding events_local and the compact tables. + #[arg(long, env = "CLICKHOUSE_DATABASE", default_value = "nostr")] + pub clickhouse_database: String, + + /// ClickHouse user. + #[arg(long, env = "CLICKHOUSE_USER")] + pub clickhouse_user: Option, + + /// ClickHouse password. + #[arg(long, env = "CLICKHOUSE_PASSWORD", hide_env_values = true)] + pub clickhouse_password: Option, +} + +impl ConnectionArgs { + /// Build a client for the configured database. + pub fn client(&self) -> Client { + let mut client = Client::default() + .with_url(&self.clickhouse_url) + .with_database(&self.clickhouse_database); + if let Some(user) = self.clickhouse_user.as_deref() { + client = client.with_user(user); + } + if let Some(password) = self.clickhouse_password.as_deref() { + client = client.with_password(password); + } + client + } +} + +/// Per-query server-side resource limits for operator queries. +#[derive(Debug, Clone, clap::Args)] +pub struct LimitArgs { + /// `max_threads` for each query. + #[arg(long, default_value_t = 2, value_parser = clap::value_parser!(u32).range(1..=32))] + pub max_threads: u32, + + /// `max_memory_usage` for each query, in GiB. Aggregation spills to disk + /// at half of this. + #[arg(long, default_value_t = 8, value_parser = clap::value_parser!(u64).range(1..=256))] + pub max_memory_gib: u64, + + /// `max_execution_time` for each query, in seconds. + #[arg(long, default_value_t = 3600, value_parser = clap::value_parser!(u64).range(1..=86400))] + pub max_execution_secs: u64, +} + +impl LimitArgs { + /// Apply the limits (and safe insert settings) to a client. + pub fn apply(&self, client: Client) -> Client { + let memory = self.max_memory_gib * GIB; + client + .with_option("max_threads", self.max_threads.to_string()) + .with_option("max_memory_usage", memory.to_string()) + .with_option( + "max_bytes_before_external_group_by", + (memory / 2).to_string(), + ) + .with_option("max_bytes_before_external_sort", (memory / 2).to_string()) + .with_option("max_execution_time", self.max_execution_secs.to_string()) + .with_option("timeout_overflow_mode", "throw") + .with_option("max_insert_threads", "1") + .with_option("async_insert", "0") + } +} + +/// A unique query id for correlating with `system.query_log`. +pub fn new_query_id(prefix: &str) -> String { + static COUNTER: AtomicU64 = AtomicU64::new(0); + let nanos = SystemTime::now() + .duration_since(UNIX_EPOCH) + .map_or(0, |d| d.as_nanos()); + let n = COUNTER.fetch_add(1, Ordering::Relaxed); + format!("{prefix}-{nanos}-{n}") +} + +/// Server clock as a Unix timestamp. +pub async fn server_now(client: &Client) -> Result { + client + .query("SELECT toUInt32(now())") + .fetch_one::() + .await + .context("reading server clock") +} + +/// Creation time of a materialized view (`metadata_modification_time`), or +/// `None` when it does not exist. +pub async fn view_created_at(client: &Client, database: &str, view: &str) -> Result> { + client + .query( + "SELECT toUInt32(metadata_modification_time) FROM system.tables \ + WHERE database = ? AND name = ?", + ) + .bind(database) + .bind(view) + .fetch_optional::() + .await + .with_context(|| format!("reading creation time of {database}.{view}")) +} + +/// Flush system logs. Ignored when the user lacks the privilege; callers poll. +pub async fn flush_logs(client: &Client) { + if let Err(e) = client.query("SYSTEM FLUSH LOGS").execute().await { + tracing::debug!(error = %e, "SYSTEM FLUSH LOGS failed; waiting for periodic flush"); + } +} + +/// Selected `system.query_log` fields for a finished query. +#[derive(Debug, Clone, Default, PartialEq, Eq, Serialize, Deserialize, clickhouse::Row)] +pub struct QueryLogEntry { + /// `type` (`QueryFinish` or an exception type). + pub kind: String, + /// Server-side duration. + pub query_duration_ms: u64, + /// Rows read. + pub read_rows: u64, + /// Bytes read. + pub read_bytes: u64, + /// Rows written (INSERT). + pub written_rows: u64, + /// Peak memory. + pub memory_usage: i64, + /// Projections used (`db.table.projection`). + pub projections: Vec, +} + +/// Fetch the final `system.query_log` entry for `query_id`, polling for up to +/// `timeout` while logs flush. Returns `None` if it never appears. +pub async fn query_log_entry( + client: &Client, + query_id: &str, + timeout: Duration, +) -> Result> { + let deadline = tokio::time::Instant::now() + timeout; + loop { + flush_logs(client).await; + let entry = client + .query( + "SELECT toString(type) AS kind, query_duration_ms, read_rows, read_bytes, \ + written_rows, memory_usage, projections \ + FROM system.query_log \ + WHERE event_date >= yesterday() AND query_id = ? AND type != 'QueryStart' \ + ORDER BY event_time_microseconds DESC LIMIT 1", + ) + .bind(query_id) + .fetch_optional::() + .await + .context("reading system.query_log")?; + if entry.is_some() || tokio::time::Instant::now() >= deadline { + return Ok(entry); + } + tokio::time::sleep(Duration::from_secs(1)).await; + } +} + +/// Space on the disk that stores a table. +#[derive(Debug, Clone, Default, PartialEq, Eq, Serialize, Deserialize, clickhouse::Row)] +pub struct DiskSpace { + /// Disk name. + pub name: String, + /// Free bytes. + pub free_space: u64, + /// Free bytes not reserved by running merges/inserts. + pub unreserved_space: u64, + /// Total bytes. + pub total_space: u64, +} + +/// Space on the disk holding `database.table` (falls back to `default`). +pub async fn table_disk(client: &Client, database: &str, table: &str) -> Result { + client + .query( + "SELECT name, free_space, unreserved_space, total_space FROM system.disks \ + WHERE name = ifNull(nullIf(( \ + SELECT any(disk_name) FROM system.parts \ + WHERE database = ? AND table = ? AND active \ + ), ''), 'default') \ + LIMIT 1", + ) + .bind(database) + .bind(table) + .fetch_optional::() + .await + .context("reading system.disks")? + .context("no disk found in system.disks") +} + +/// Active-part statistics for a table. +#[derive(Debug, Clone, Default, PartialEq, Eq, Serialize, Deserialize, clickhouse::Row)] +pub struct TableParts { + /// Active parts. + pub parts: u64, + /// Rows across active parts (pre-merge duplicates included). + pub rows: u64, + /// Bytes on disk across active parts. + pub bytes_on_disk: u64, + /// Uncompressed data bytes. + pub data_uncompressed_bytes: u64, +} + +/// Active-part statistics for `database.table`. +pub async fn table_parts(client: &Client, database: &str, table: &str) -> Result { + client + .query( + "SELECT count() AS parts, sum(rows) AS rows, sum(bytes_on_disk) AS bytes_on_disk, \ + sum(data_uncompressed_bytes) AS data_uncompressed_bytes \ + FROM system.parts WHERE database = ? AND table = ? AND active", + ) + .bind(database) + .bind(table) + .fetch_one::() + .await + .with_context(|| format!("reading system.parts for {database}.{table}")) +} + +/// Compression ratio (compressed / uncompressed) of a column, if known. +pub async fn column_compression_ratio( + client: &Client, + database: &str, + table: &str, + column: &str, +) -> Result> { + let ratio = client + .query( + "SELECT if(data_uncompressed_bytes = 0, -1, \ + data_compressed_bytes / data_uncompressed_bytes) \ + FROM system.columns WHERE database = ? AND table = ? AND name = ?", + ) + .bind(database) + .bind(table) + .bind(column) + .fetch_optional::() + .await + .context("reading system.columns")?; + Ok(ratio.filter(|r| *r > 0.0)) +} + +/// Inserts into `events_local` that started before `t0` and are still +/// running (they may have built their pipeline before a view existed). +pub async fn inserts_started_before(client: &Client, database: &str, t0: u32) -> Result { + let now = server_now(client).await?; + let started_before_secs = now.saturating_sub(t0); + client + .query( + "SELECT count() FROM system.processes \ + WHERE query_kind = 'Insert' AND positionCaseInsensitive(query, 'events_local') > 0 \ + AND (current_database = ? OR positionCaseInsensitive(query, concat(?, '.')) > 0) \ + AND elapsed >= ?", + ) + .bind(database) + .bind(database) + .bind(started_before_secs as f64) + .fetch_one::() + .await + .context("reading system.processes") +} + +/// Pending async-insert buffers for `events_local` first written before `t0`. +pub async fn async_inserts_queued_before(client: &Client, database: &str, t0: u32) -> Result { + client + .query( + "SELECT count() FROM system.asynchronous_inserts \ + WHERE database = ? AND table = 'events_local' \ + AND toUInt32(first_update) < ?", + ) + .bind(database) + .bind(t0) + .fetch_one::() + .await + .context("reading system.asynchronous_inserts") +} + +/// Fail unless the target's materialized view exists; returns its creation +/// time. +pub async fn require_view(client: &Client, database: &str, view: &str) -> Result { + match view_created_at(client, database, view).await? { + Some(t0) => Ok(t0), + None => bail!( + "{database}.{view} does not exist; apply \ + docs/migrations/prepared/017_preview_compact_lookups.sql first" + ), + } +} + +/// Human-readable byte size. +pub fn human_bytes(bytes: u64) -> String { + const UNITS: [&str; 6] = ["B", "KiB", "MiB", "GiB", "TiB", "PiB"]; + let mut value = bytes as f64; + let mut unit = 0; + while value >= 1024.0 && unit < UNITS.len() - 1 { + value /= 1024.0; + unit += 1; + } + if unit == 0 { + format!("{bytes} B") + } else { + format!("{value:.2} {}", UNITS[unit]) + } +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn human_bytes_units() { + assert_eq!(human_bytes(0), "0 B"); + assert_eq!(human_bytes(1023), "1023 B"); + assert_eq!(human_bytes(1024), "1.00 KiB"); + assert_eq!(human_bytes(GIB * 3 / 2), "1.50 GiB"); + assert_eq!(human_bytes(GIB * 1024 * 2), "2.00 TiB"); + } + + #[test] + fn query_ids_are_unique() { + let a = new_query_id("x"); + let b = new_query_id("x"); + assert_ne!(a, b); + assert!(a.starts_with("x-")); + } +} diff --git a/crates/pensieve-preview/src/lookups/oracle.rs b/crates/pensieve-preview/src/lookups/oracle.rs new file mode 100644 index 0000000..3c7605c --- /dev/null +++ b/crates/pensieve-preview/src/lookups/oracle.rs @@ -0,0 +1,370 @@ +//! Independent Rust reference implementations of the compact lookup rules. +//! +//! Validation and tests compare ClickHouse results against these functions +//! computed from the raw source events, so a bug in the SQL cannot hide behind +//! the same bug in the expected values. + +use std::collections::{BTreeMap, HashMap}; +use std::sync::LazyLock; + +use regex::Regex; + +use crate::query::ZapTotals; + +/// Zap amounts above this many msats are treated as fake and counted as 0 +/// (migration 009). +pub const MAX_ZAP_AMOUNT_MSATS: u64 = 1_000_000_000; + +/// A source event as stored in `events_local`. +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct SourceEvent { + /// Event id (hex). + pub id: String, + /// Author pubkey (hex). + pub pubkey: String, + /// Unix `created_at`. + pub created_at: u32, + /// Event kind. + pub kind: u16, + /// Event content. + pub content: String, + /// Event tags. + pub tags: Vec>, +} + +/// Pick the replaceable-event winner: highest `created_at`, then lowest id +/// (NIP-01). Duplicates of the same event are harmless. +pub fn replaceable_winner<'a>( + events: impl IntoIterator, +) -> Option<&'a SourceEvent> { + events.into_iter().min_by(|a, b| { + b.created_at + .cmp(&a.created_at) + .then_with(|| a.id.cmp(&b.id)) + }) +} + +/// Number of distinct event ids sharing the winning `created_at`. +pub fn tie_width<'a>(events: impl IntoIterator + Clone) -> usize { + let Some(max) = events.clone().into_iter().map(|e| e.created_at).max() else { + return 0; + }; + let mut ids: Vec<&str> = events + .into_iter() + .filter(|e| e.created_at == max) + .map(|e| e.id.as_str()) + .collect(); + ids.sort_unstable(); + ids.dedup(); + ids.len() +} + +/// First `d` tag value, matching the `events_local.d_tag` expression +/// (`''` when absent or valueless). +pub fn d_tag(tags: &[Vec]) -> &str { + tags.iter() + .find(|t| t.first().map(String::as_str) == Some("d")) + .and_then(|t| t.get(1)) + .map_or("", String::as_str) +} + +/// Distinct non-empty `e` tag values, in first-seen order. +pub fn zap_targets(tags: &[Vec]) -> Vec { + let mut targets: Vec = Vec::new(); + for tag in tags { + if tag.len() >= 2 && tag[0] == "e" && !tag[1].is_empty() && !targets.contains(&tag[1]) { + targets.push(tag[1].clone()); + } + } + targets +} + +static MULTIPLIERS: LazyLock<[(Regex, Multiplier); 4]> = LazyLock::new(|| { + [ + (re(r"^lnbc([0-9]+)m1"), Multiplier::Mul(100_000_000)), + (re(r"^lnbc([0-9]+)u1"), Multiplier::Mul(100_000)), + (re(r"^lnbc([0-9]+)n1"), Multiplier::Mul(100)), + (re(r"^lnbc([0-9]+)p1"), Multiplier::Div(10)), + ] +}); + +static PARSEABLE: LazyLock = LazyLock::new(|| re(r"^lnbc[0-9]+[munp]1")); + +fn re(pattern: &str) -> Regex { + Regex::new(pattern).expect("static regex") +} + +#[derive(Clone, Copy)] +enum Multiplier { + Mul(u64), + Div(u64), +} + +/// Parse a zap receipt amount with the migration 009 rules. +/// +/// Returns `(amount_msats, parsed)`: +/// - the amount comes from the *first* `bolt11` tag, using the first matching +/// multiplier in m, u, n, p order; integer overflow follows ClickHouse +/// (`toUInt64OrZero` gives 0 and multiplication wraps); +/// - amounts above [`MAX_ZAP_AMOUNT_MSATS`] become 0; +/// - `parsed` is true when *any* `bolt11` tag matches `^lnbc[0-9]+[munp]1`, +/// which is the legacy inclusion rule for counting a receipt. +pub fn zap_amount(tags: &[Vec]) -> (u64, bool) { + let first_bolt11 = tags + .iter() + .find(|t| t.first().map(String::as_str) == Some("bolt11")) + .and_then(|t| t.get(1)) + .map_or("", String::as_str); + + let mut raw = 0u64; + for (regex, multiplier) in MULTIPLIERS.iter() { + let digits = regex + .captures(first_bolt11) + .and_then(|c| c.get(1)) + .map_or("", |m| m.as_str()); + if digits.is_empty() { + continue; + } + let value = digits.parse::().unwrap_or(0); + raw = match multiplier { + Multiplier::Mul(factor) => value.wrapping_mul(*factor), + Multiplier::Div(divisor) => value / divisor, + }; + break; + } + let amount = if raw > MAX_ZAP_AMOUNT_MSATS { 0 } else { raw }; + + let parsed = tags + .iter() + .any(|t| t.len() >= 2 && t[0] == "bolt11" && PARSEABLE.is_match(&t[1])); + + (amount, parsed) +} + +/// Expected zap totals per target from a set of kind 9735 receipts. +/// +/// Receipts are deduplicated by id, so replays never inflate totals. +pub fn zap_totals_by_target<'a>( + receipts: impl IntoIterator, +) -> HashMap { + let mut unique: BTreeMap<&str, &SourceEvent> = BTreeMap::new(); + for receipt in receipts { + if receipt.kind == 9735 { + unique.entry(receipt.id.as_str()).or_insert(receipt); + } + } + + let mut totals: HashMap = HashMap::new(); + for receipt in unique.values() { + let (amount, parsed) = zap_amount(&receipt.tags); + if !parsed { + continue; + } + for target in zap_targets(&receipt.tags) { + let entry = totals.entry(target).or_default(); + entry.zap_count += 1; + entry.total_msats = entry.total_msats.wrapping_add(amount); + } + } + totals +} + +/// Expected latest profile per pubkey from kind 0 source events. +pub fn latest_profiles<'a>( + events: impl IntoIterator, +) -> HashMap { + let mut grouped: HashMap<&str, Vec<&SourceEvent>> = HashMap::new(); + for event in events { + if event.kind == 0 { + grouped + .entry(event.pubkey.as_str()) + .or_default() + .push(event); + } + } + grouped + .into_iter() + .filter_map(|(pubkey, events)| { + replaceable_winner(events).map(|winner| (pubkey.to_string(), winner)) + }) + .collect() +} + +/// Expected latest event per (pubkey, kind, d_tag) for kinds covered by +/// `addressable_latest`. +pub fn latest_addressable<'a>( + events: impl IntoIterator, +) -> HashMap<(String, u16, String), &'a SourceEvent> { + let mut grouped: HashMap<(String, u16, String), Vec<&SourceEvent>> = HashMap::new(); + for event in events { + if super::compact::addressable_kind_covered(event.kind) { + let key = ( + event.pubkey.clone(), + event.kind, + d_tag(&event.tags).to_string(), + ); + grouped.entry(key).or_default().push(event); + } + } + grouped + .into_iter() + .filter_map(|(key, events)| replaceable_winner(events).map(|winner| (key, winner))) + .collect() +} + +#[cfg(test)] +mod tests { + use super::*; + + fn tags(raw: &[&[&str]]) -> Vec> { + raw.iter() + .map(|t| t.iter().map(|s| s.to_string()).collect()) + .collect() + } + + fn event(id: &str, created_at: u32) -> SourceEvent { + SourceEvent { + id: id.to_string(), + pubkey: "p".to_string(), + created_at, + kind: 0, + content: id.to_string(), + tags: Vec::new(), + } + } + + #[test] + fn winner_prefers_newest_then_lowest_id() { + let events = [ + event("b", 10), + event("a", 9), + event("c", 10), + event("b", 10), + ]; + assert_eq!(replaceable_winner(&events).unwrap().id, "b"); + assert_eq!(tie_width(&events), 2); + + let reversed: Vec<_> = events.iter().rev().cloned().collect(); + assert_eq!(replaceable_winner(&reversed).unwrap().id, "b"); + } + + #[test] + fn winner_of_empty_is_none() { + let none: [SourceEvent; 0] = []; + assert!(replaceable_winner(&none).is_none()); + assert_eq!(tie_width(&none), 0); + } + + #[test] + fn d_tag_matches_clickhouse_first_d_semantics() { + assert_eq!( + d_tag(&tags(&[&["t", "x"], &["d", "slug"], &["d", "other"]])), + "slug" + ); + assert_eq!(d_tag(&tags(&[&["d"], &["d", "slug"]])), ""); + assert_eq!(d_tag(&tags(&[])), ""); + } + + #[test] + fn zap_targets_are_distinct_and_non_empty() { + let t = tags(&[ + &["e", "a"], + &["e"], + &["e", ""], + &["e", "a"], + &["e", "b", "relay"], + &[], + ]); + assert_eq!(zap_targets(&t), vec!["a".to_string(), "b".to_string()]); + } + + #[test] + fn zap_amount_multipliers() { + assert_eq!( + zap_amount(&tags(&[&["bolt11", "lnbc2m1xyz"]])), + (200_000_000, true) + ); + assert_eq!( + zap_amount(&tags(&[&["bolt11", "lnbc21u1xyz"]])), + (2_100_000, true) + ); + assert_eq!( + zap_amount(&tags(&[&["bolt11", "lnbc210n1xyz"]])), + (21_000, true) + ); + assert_eq!( + zap_amount(&tags(&[&["bolt11", "lnbc2105p1xyz"]])), + (210, true) + ); + } + + #[test] + fn zap_amount_limits_and_rejections() { + // Above 1B msats is zeroed but still counted. + assert_eq!(zap_amount(&tags(&[&["bolt11", "lnbc11m1x"]])), (0, true)); + // Exactly 1B msats is kept. + assert_eq!( + zap_amount(&tags(&[&["bolt11", "lnbc10m1x"]])), + (1_000_000_000, true) + ); + // No multiplier: not parsed. + assert_eq!(zap_amount(&tags(&[&["bolt11", "lnbc1500x"]])), (0, false)); + // Testnet invoices are not parsed. + assert_eq!(zap_amount(&tags(&[&["bolt11", "lntb10u1x"]])), (0, false)); + // u64 overflow in the digits gives 0 (toUInt64OrZero). + assert_eq!( + zap_amount(&tags(&[&["bolt11", "lnbc99999999999999999999999m1x"]])), + (0, true) + ); + // Missing / valueless bolt11. + assert_eq!(zap_amount(&tags(&[&["e", "x"]])), (0, false)); + assert_eq!(zap_amount(&tags(&[&["bolt11"]])), (0, false)); + } + + #[test] + fn zap_amount_uses_first_bolt11_but_any_for_inclusion() { + // First bolt11 is garbage, second is valid: counted with amount 0. + let t = tags(&[&["bolt11", "garbage"], &["bolt11", "lnbc5u1x"]]); + assert_eq!(zap_amount(&t), (0, true)); + // First valueless bolt11 also shadows later ones for the amount. + let t = tags(&[&["bolt11"], &["bolt11", "lnbc5u1x"]]); + assert_eq!(zap_amount(&t), (0, true)); + } + + #[test] + fn zap_totals_dedupe_replays() { + let receipt = SourceEvent { + id: "r1".to_string(), + pubkey: "z".to_string(), + created_at: 1, + kind: 9735, + content: String::new(), + tags: tags(&[&["bolt11", "lnbc10n1x"], &["e", "t"], &["e", "t"]]), + }; + let other = SourceEvent { + id: "r2".to_string(), + tags: tags(&[&["bolt11", "lnbc20n1x"], &["e", "t"], &["e", "u"]]), + ..receipt.clone() + }; + let unparsed = SourceEvent { + id: "r3".to_string(), + tags: tags(&[&["bolt11", "nope"], &["e", "t"]]), + ..receipt.clone() + }; + let totals = zap_totals_by_target([&receipt, &receipt, &other, &unparsed, &other]); + assert_eq!( + totals["t"], + ZapTotals { + total_msats: 3_000, + zap_count: 2 + } + ); + assert_eq!( + totals["u"], + ZapTotals { + total_msats: 2_000, + zap_count: 1 + } + ); + } +} diff --git a/crates/pensieve-preview/src/lookups/plan.rs b/crates/pensieve-preview/src/lookups/plan.rs new file mode 100644 index 0000000..a7bee07 --- /dev/null +++ b/crates/pensieve-preview/src/lookups/plan.rs @@ -0,0 +1,668 @@ +//! Backfill shard plans, disk estimates and on-disk checkpoint layout. +//! +//! A plan splits one target's source rows into contiguous shards. The first +//! and last `created_at` shards are open-ended, so every row is covered +//! regardless of timestamp (including `created_at = 0` and future values). +//! Plans and per-shard checkpoints are JSON files under +//! `//`, which makes backfills resumable. + +use std::path::{Path, PathBuf}; + +use anyhow::{Context, Result, bail}; +use clickhouse::Client; +use serde::{Deserialize, Serialize}; + +use super::ops::{self, GIB}; +use super::target::Target; + +/// Plan file format version. +pub const PLAN_FORMAT_VERSION: u32 = 1; + +/// Assumed compression ratio for random hex and fixed-width columns. +const HEX_COMPRESSION_RATIO: f64 = 0.6; + +/// Fallback compression ratio for free text when `system.columns` has none. +const TEXT_COMPRESSION_RATIO: f64 = 0.5; + +/// How shards partition the source rows. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize, clap::ValueEnum)] +#[serde(rename_all = "kebab-case")] +pub enum ShardBy { + /// `created_at` windows; reads only the target kinds through the + /// `events_by_kind` projection. Use while the projection exists. + CreatedAt, + /// Event id prefixes; each shard reads a primary-key range of the full + /// table. Only for rebuilds after the projection has been removed. + IdPrefix, +} + +/// The source-row range a shard covers. +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] +#[serde(tag = "by", rename_all = "snake_case")] +pub enum ShardRange { + /// `start <= created_at < end`; `None` is unbounded. + CreatedAt { + /// Inclusive lower bound (Unix seconds). + start: Option, + /// Exclusive upper bound (Unix seconds). + end: Option, + }, + /// `start <= id < end`; `None` is unbounded. Bounds are lowercase hex. + IdPrefix { + /// Inclusive lower bound. + start: Option, + /// Exclusive upper bound. + end: Option, + }, + /// `start <= indexed_at <= end`, used by the gap-closure pass. + IndexedAt { + /// Inclusive lower bound (Unix seconds). + start: u32, + /// Inclusive upper bound (Unix seconds). + end: u32, + }, +} + +impl ShardRange { + /// SQL predicate for this range. Values are integers or validated hex, so + /// the result is safe to embed. + pub fn predicate(&self) -> String { + let mut parts = Vec::new(); + match self { + Self::CreatedAt { start, end } => { + if let Some(start) = start { + parts.push(format!("created_at >= toDateTime({start})")); + } + if let Some(end) = end { + parts.push(format!("created_at < toDateTime({end})")); + } + } + Self::IdPrefix { start, end } => { + if let Some(start) = start { + assert!(is_hex(start), "id prefix bound must be lowercase hex"); + parts.push(format!("id >= '{start}'")); + } + if let Some(end) = end { + assert!(is_hex(end), "id prefix bound must be lowercase hex"); + parts.push(format!("id < '{end}'")); + } + } + Self::IndexedAt { start, end } => { + parts.push(format!("indexed_at >= toDateTime({start})")); + parts.push(format!("indexed_at <= toDateTime({end})")); + } + } + if parts.is_empty() { + "1".to_string() + } else { + parts.join(" AND ") + } + } +} + +fn is_hex(value: &str) -> bool { + value + .bytes() + .all(|b| b.is_ascii_digit() || (b'a'..=b'f').contains(&b)) +} + +/// One unit of backfill work. +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] +pub struct Shard { + /// Position in the plan. + pub index: usize, + /// Source rows covered. + pub range: ShardRange, + /// Source rows expected in the range at planning time (0 if unknown). + pub estimated_rows: u64, +} + +/// Split a per-hour source histogram into contiguous `created_at` shards of +/// at most `max_rows` rows (a single hour larger than `max_rows` is its own +/// shard). The first shard has no lower bound and the last no upper bound. +pub fn shards_from_histogram(histogram: &[(u32, u64)], max_rows: u64) -> Vec { + let max_rows = max_rows.max(1); + let mut boundaries: Vec = Vec::new(); + let mut rows_per_shard: Vec = vec![0]; + + for &(bucket, rows) in histogram { + let current = *rows_per_shard.last().expect("non-empty"); + if current > 0 && current.saturating_add(rows) > max_rows { + boundaries.push(bucket); + rows_per_shard.push(0); + } + *rows_per_shard.last_mut().expect("non-empty") += rows; + } + + let mut shards = Vec::with_capacity(rows_per_shard.len()); + for (index, estimated_rows) in rows_per_shard.into_iter().enumerate() { + let start = index.checked_sub(1).map(|i| boundaries[i]); + let end = boundaries.get(index).copied(); + shards.push(Shard { + index, + range: ShardRange::CreatedAt { start, end }, + estimated_rows, + }); + } + shards +} + +/// Contiguous id-prefix shards over `16^prefix_len` buckets, with open-ended +/// first and last shards. +pub fn id_prefix_shards(prefix_len: usize, total_rows: u64) -> Vec { + assert!((1..=3).contains(&prefix_len), "prefix length must be 1..=3"); + let buckets = 16usize.pow(prefix_len as u32); + let per_shard = total_rows / buckets as u64; + (0..buckets) + .map(|index| { + let bound = |i: usize| format!("{i:0width$x}", width = prefix_len); + Shard { + index, + range: ShardRange::IdPrefix { + start: (index > 0).then(|| bound(index)), + end: (index + 1 < buckets).then(|| bound(index + 1)), + }, + estimated_rows: per_shard, + } + }) + .collect() +} + +/// Source statistics for a target. +#[derive(Debug, Clone, Default, PartialEq, Serialize, Deserialize, clickhouse::Row)] +pub struct SourceStats { + /// Source rows in `events_local` (pre-merge duplicates included). + pub source_rows: u64, + /// Approximate distinct keys (pubkeys, receipts or coordinates). + pub distinct_keys: u64, + /// Approximate rows in the merged target table. + pub output_rows: u64, + /// Mean free-text payload bytes per output row (content or d tag). + pub avg_payload_bytes: f64, +} + +/// Disk estimate for a backfill. +#[derive(Debug, Clone, Default, PartialEq, Serialize, Deserialize)] +pub struct DiskEstimate { + /// Estimated compressed bytes per target row. + pub bytes_per_row: u64, + /// Rows written by the backfill before merges (states/rows across shards). + pub pre_merge_rows: u64, + /// Estimated size after merges. + pub final_bytes: u64, + /// Estimated size written by the backfill before merges. + pub pre_merge_bytes: u64, + /// Peak disk: pre-merge bytes plus equal merge headroom (a merge writes + /// its output while its inputs, then outdated parts, still exist). + pub peak_bytes: u64, + /// Compression ratio used for free text. + pub text_compression_ratio: f64, +} + +/// Estimate disk use for backfilling `target`. +pub fn estimate_disk( + target: Target, + stats: &SourceStats, + shards: &[Shard], + text_compression_ratio: f64, +) -> DiskEstimate { + let (fixed, payload) = target.row_bytes(stats.avg_payload_bytes); + let bytes_per_row = + (fixed * HEX_COMPRESSION_RATIO + payload * text_compression_ratio).ceil() as u64; + + let pre_merge_rows = if target.aggregates_per_key() { + // A shard writes at most one state per key it contains. + shards + .iter() + .map(|s| s.estimated_rows.min(stats.distinct_keys)) + .sum::() + .max(stats.output_rows) + } else { + stats.output_rows + }; + + let final_bytes = stats.output_rows.saturating_mul(bytes_per_row); + let pre_merge_bytes = pre_merge_rows.saturating_mul(bytes_per_row); + DiskEstimate { + bytes_per_row, + pre_merge_rows, + final_bytes, + pre_merge_bytes, + peak_bytes: pre_merge_bytes.saturating_mul(2), + text_compression_ratio, + } +} + +/// A persisted backfill plan for one target. +#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)] +pub struct Plan { + /// File format version. + pub format_version: u32, + /// Database containing `events_local` and the target. + pub database: String, + /// Target table. + pub target: Target, + /// Shard dimension. + pub shard_by: ShardBy, + /// Whether every shard must be served by the `events_by_kind` projection. + pub require_projection: bool, + /// Creation time (Unix seconds) of the target's materialized view. Rows + /// inserted after this are captured by the view. + pub view_created_at: u32, + /// Server time when the plan was made. + pub planned_at: u32, + /// Source statistics at planning time. + pub source: SourceStats, + /// Disk estimate. + pub estimate: DiskEstimate, + /// Shards in execution order. + pub shards: Vec, +} + +/// Options for [`build_plan`]. +#[derive(Debug, Clone)] +pub struct PlanOptions { + /// Shard dimension. + pub shard_by: ShardBy, + /// Target source rows per `created_at` shard. + pub max_rows_per_shard: u64, + /// Hex prefix length for id-prefix shards. + pub id_prefix_len: usize, + /// Require the `events_by_kind` projection for planning and backfill. + pub require_projection: bool, +} + +/// Apply projection settings for queries that must read `events_by_kind`. +pub fn projection_client(client: &Client, require_projection: bool) -> Client { + if require_projection { + client + .clone() + .with_option("optimize_use_projections", "1") + .with_option("force_optimize_projection", "1") + } else { + client.clone() + } +} + +/// Inspect the source and build a plan for `target`. +pub async fn build_plan( + client: &Client, + database: &str, + target: Target, + options: &PlanOptions, +) -> Result { + if options.shard_by == ShardBy::IdPrefix && options.require_projection { + bail!("id-prefix shards read the base table; pass --allow-full-scan"); + } + + let view_created_at = ops::require_view(client, database, target.materialized_view()).await?; + let planned_at = ops::server_now(client).await?; + let source_client = projection_client(client, options.require_projection); + + let source = source_client + .query(&target.source_stats_sql(database)) + .fetch_one::() + .await + .with_context(|| format!("reading {target} source statistics"))?; + + let shards = match options.shard_by { + ShardBy::CreatedAt => { + let histogram = source_client + .query(&target.hour_histogram_sql(database)) + .fetch_all::<(u32, u64)>() + .await + .with_context(|| format!("reading {target} hour histogram"))?; + shards_from_histogram(&histogram, options.max_rows_per_shard) + } + ShardBy::IdPrefix => id_prefix_shards(options.id_prefix_len, source.source_rows), + }; + + let text_ratio = ops::column_compression_ratio(client, database, "events_local", "content") + .await? + .unwrap_or(TEXT_COMPRESSION_RATIO); + let estimate = estimate_disk(target, &source, &shards, text_ratio); + + Ok(Plan { + format_version: PLAN_FORMAT_VERSION, + database: database.to_string(), + target, + shard_by: options.shard_by, + require_projection: options.require_projection, + view_created_at, + planned_at, + source, + estimate, + shards, + }) +} + +/// Result of one completed shard, persisted as its checkpoint. +#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)] +pub struct ShardCheckpoint { + /// Shard index. + pub index: usize, + /// Shard range. + pub range: ShardRange, + /// ClickHouse query id of the INSERT. + pub query_id: String, + /// Client-observed duration. + pub elapsed_ms: u64, + /// `system.query_log` entry, when available. + pub query_log: Option, + /// Server time when the shard finished. + pub finished_at: u32, +} + +/// Checkpoint directory layout for one target. +#[derive(Debug, Clone)] +pub struct CheckpointDir { + root: PathBuf, +} + +impl CheckpointDir { + /// Layout rooted at `/`. + pub fn new(base: &Path, target: Target) -> Self { + Self { + root: base.join(target.name()), + } + } + + /// Directory for this target. + pub fn root(&self) -> &Path { + &self.root + } + + /// Plan file path. + pub fn plan_path(&self) -> PathBuf { + self.root.join("plan.json") + } + + fn shard_path(&self, index: usize) -> PathBuf { + self.root.join("shards").join(format!("{index:06}.json")) + } + + /// Gap-closure pass checkpoint path. + pub fn gap_pass_path(&self) -> PathBuf { + self.root.join("gap-pass.json") + } + + /// Load the plan, if present. + pub fn load_plan(&self) -> Result> { + read_json(&self.plan_path()) + } + + /// Persist the plan. Refuses to replace a plan with completed shards + /// unless `discard_progress` is set (which also removes them). + pub fn save_plan(&self, plan: &Plan, discard_progress: bool) -> Result<()> { + let done = self.completed_shards()?; + if !done.is_empty() { + if !discard_progress { + bail!( + "{} already has {} completed shard(s); pass --discard-progress to replan", + self.root.display(), + done.len() + ); + } + std::fs::remove_dir_all(self.root.join("shards")) + .with_context(|| format!("removing {}", self.root.display()))?; + } + write_json_atomic(&self.plan_path(), plan) + } + + /// Record a completed shard. + pub fn save_shard(&self, checkpoint: &ShardCheckpoint) -> Result<()> { + write_json_atomic(&self.shard_path(checkpoint.index), checkpoint) + } + + /// Record the gap-closure pass. + pub fn save_gap_pass(&self, checkpoint: &ShardCheckpoint) -> Result<()> { + write_json_atomic(&self.gap_pass_path(), checkpoint) + } + + /// Completed shard checkpoints, ordered by index. + pub fn completed_shards(&self) -> Result> { + let dir = self.root.join("shards"); + let entries = match std::fs::read_dir(&dir) { + Ok(entries) => entries, + Err(e) if e.kind() == std::io::ErrorKind::NotFound => return Ok(Vec::new()), + Err(e) => return Err(e).with_context(|| format!("reading {}", dir.display())), + }; + let mut done = Vec::new(); + for entry in entries { + let path = entry?.path(); + if path.extension().and_then(|e| e.to_str()) == Some("json") + && let Some(checkpoint) = read_json::(&path)? + { + done.push(checkpoint); + } + } + done.sort_by_key(|c| c.index); + Ok(done) + } +} + +fn read_json(path: &Path) -> Result> { + match std::fs::read(path) { + Ok(bytes) => serde_json::from_slice(&bytes) + .map(Some) + .with_context(|| format!("parsing {}", path.display())), + Err(e) if e.kind() == std::io::ErrorKind::NotFound => Ok(None), + Err(e) => Err(e).with_context(|| format!("reading {}", path.display())), + } +} + +fn write_json_atomic(path: &Path, value: &T) -> Result<()> { + let parent = path.parent().context("checkpoint path has no parent")?; + std::fs::create_dir_all(parent).with_context(|| format!("creating {}", parent.display()))?; + let tmp = path.with_extension("json.tmp"); + let bytes = serde_json::to_vec_pretty(value)?; + std::fs::write(&tmp, bytes).with_context(|| format!("writing {}", tmp.display()))?; + std::fs::rename(&tmp, path).with_context(|| format!("renaming to {}", path.display()))?; + Ok(()) +} + +/// Print a plan summary for operators. +pub fn describe_plan(plan: &Plan, min_free_bytes: u64) -> String { + let e = &plan.estimate; + let required = e.peak_bytes.saturating_add(min_free_bytes); + format!( + "target {target} ({table}) in {db}\n\ + \x20 source rows: {source_rows}\n\ + \x20 distinct keys: {keys}\n\ + \x20 target rows (merged): {output}\n\ + \x20 shards: {shards} by {by:?} (projection required: {proj})\n\ + \x20 bytes per row (est): {bpr}\n\ + \x20 final size (est): {final_size}\n\ + \x20 pre-merge size (est): {pre} ({pre_rows} rows)\n\ + \x20 peak disk (est): {peak} = pre-merge x2 for merge headroom\n\ + \x20 free space needed: {required} including {reserve} reserve\n\ + \x20 view created at: {t0}", + target = plan.target, + table = plan.target.table(), + db = plan.database, + source_rows = plan.source.source_rows, + keys = plan.source.distinct_keys, + output = plan.source.output_rows, + shards = plan.shards.len(), + by = plan.shard_by, + proj = plan.require_projection, + bpr = e.bytes_per_row, + final_size = ops::human_bytes(e.final_bytes), + pre = ops::human_bytes(e.pre_merge_bytes), + pre_rows = e.pre_merge_rows, + peak = ops::human_bytes(e.peak_bytes), + required = ops::human_bytes(required), + reserve = ops::human_bytes(min_free_bytes), + t0 = plan.view_created_at, + ) +} + +/// Default free-space reserve kept untouched during backfill. +pub const DEFAULT_MIN_FREE_BYTES: u64 = 100 * GIB; + +#[cfg(test)] +mod tests { + use super::*; + + fn bounds(shards: &[Shard]) -> Vec<(Option, Option)> { + shards + .iter() + .map(|s| match &s.range { + ShardRange::CreatedAt { start, end } => (*start, *end), + other => panic!("unexpected {other:?}"), + }) + .collect() + } + + #[test] + fn empty_histogram_is_one_unbounded_shard() { + let shards = shards_from_histogram(&[], 10); + assert_eq!(bounds(&shards), vec![(None, None)]); + assert_eq!(shards[0].range.predicate(), "1"); + } + + #[test] + fn histogram_shards_are_contiguous_and_open_ended() { + let histogram = [(0, 3), (100, 4), (200, 5), (300, 1), (400, 9), (500, 2)]; + let shards = shards_from_histogram(&histogram, 8); + let b = bounds(&shards); + assert_eq!(b.first().unwrap().0, None); + assert_eq!(b.last().unwrap().1, None); + for pair in b.windows(2) { + assert_eq!(pair[0].1, pair[1].0, "shards must be contiguous"); + } + let total: u64 = shards.iter().map(|s| s.estimated_rows).sum(); + assert_eq!(total, 24); + assert!(shards.iter().all(|s| s.estimated_rows <= 9)); + assert_eq!( + b, + vec![ + (None, Some(200)), + (Some(200), Some(400)), + (Some(400), Some(500)), + (Some(500), None) + ] + ); + for (i, s) in shards.iter().enumerate() { + assert_eq!(s.index, i); + } + } + + #[test] + fn oversized_bucket_is_its_own_shard() { + let shards = shards_from_histogram(&[(1, 2), (2, 50), (3, 2)], 10); + assert_eq!( + shards.iter().map(|s| s.estimated_rows).collect::>(), + vec![2, 50, 2] + ); + } + + #[test] + fn created_at_predicates() { + let r = ShardRange::CreatedAt { + start: Some(5), + end: Some(9), + }; + assert_eq!( + r.predicate(), + "created_at >= toDateTime(5) AND created_at < toDateTime(9)" + ); + let r = ShardRange::CreatedAt { + start: None, + end: Some(9), + }; + assert_eq!(r.predicate(), "created_at < toDateTime(9)"); + let r = ShardRange::IndexedAt { start: 1, end: 2 }; + assert_eq!( + r.predicate(), + "indexed_at >= toDateTime(1) AND indexed_at <= toDateTime(2)" + ); + } + + #[test] + fn id_prefix_shards_cover_keyspace() { + let shards = id_prefix_shards(1, 160); + assert_eq!(shards.len(), 16); + assert_eq!(shards[0].range.predicate(), "id < '1'"); + assert_eq!(shards[10].range.predicate(), "id >= 'a' AND id < 'b'"); + assert_eq!(shards[15].range.predicate(), "id >= 'f'"); + assert!(shards.iter().all(|s| s.estimated_rows == 10)); + + let shards = id_prefix_shards(2, 0); + assert_eq!(shards.len(), 256); + assert_eq!(shards[255].range.predicate(), "id >= 'ff'"); + for pair in shards.windows(2) { + let (ShardRange::IdPrefix { end, .. }, ShardRange::IdPrefix { start, .. }) = + (&pair[0].range, &pair[1].range) + else { + panic!("unexpected range"); + }; + assert_eq!(end, start); + } + } + + #[test] + fn disk_estimate_counts_pre_merge_states_and_headroom() { + let stats = SourceStats { + source_rows: 1_000, + distinct_keys: 100, + output_rows: 100, + avg_payload_bytes: 200.0, + }; + let shards = shards_from_histogram(&[(1, 400), (2, 30), (3, 570)], 450); + assert_eq!(shards.len(), 2); + let e = estimate_disk(Target::Profiles, &stats, &shards, 0.5); + // Each shard holds at most one state per key: 100 + 100. + assert_eq!(e.pre_merge_rows, 200); + assert_eq!(e.final_bytes, 100 * e.bytes_per_row); + assert_eq!(e.pre_merge_bytes, 200 * e.bytes_per_row); + assert_eq!(e.peak_bytes, 2 * e.pre_merge_bytes); + + let zap_stats = SourceStats { + source_rows: 50, + distinct_keys: 50, + output_rows: 60, + avg_payload_bytes: 0.0, + }; + let e = estimate_disk(Target::Zaps, &zap_stats, &shards, 0.5); + assert_eq!(e.pre_merge_rows, 60); + } + + #[test] + fn checkpoint_dir_round_trip_and_replan_guard() { + let dir = tempfile::tempdir().unwrap(); + let cp = CheckpointDir::new(dir.path(), Target::Zaps); + assert!(cp.load_plan().unwrap().is_none()); + assert!(cp.completed_shards().unwrap().is_empty()); + + let plan = Plan { + format_version: PLAN_FORMAT_VERSION, + database: "db".into(), + target: Target::Zaps, + shard_by: ShardBy::CreatedAt, + require_projection: true, + view_created_at: 7, + planned_at: 8, + source: SourceStats::default(), + estimate: DiskEstimate::default(), + shards: shards_from_histogram(&[(1, 1), (2, 1)], 1), + }; + cp.save_plan(&plan, false).unwrap(); + assert_eq!(cp.load_plan().unwrap().unwrap(), plan); + + let checkpoint = ShardCheckpoint { + index: 1, + range: plan.shards[1].range.clone(), + query_id: "q".into(), + elapsed_ms: 3, + query_log: None, + finished_at: 9, + }; + cp.save_shard(&checkpoint).unwrap(); + assert_eq!(cp.completed_shards().unwrap(), vec![checkpoint]); + + assert!(cp.save_plan(&plan, false).is_err()); + cp.save_plan(&plan, true).unwrap(); + assert!(cp.completed_shards().unwrap().is_empty()); + } +} diff --git a/crates/pensieve-preview/src/lookups/status.rs b/crates/pensieve-preview/src/lookups/status.rs new file mode 100644 index 0000000..3288026 --- /dev/null +++ b/crates/pensieve-preview/src/lookups/status.rs @@ -0,0 +1,102 @@ +//! Backfill progress, table sizes and freshness for the compact tables. + +use std::path::Path; + +use anyhow::Result; +use clickhouse::Client; +use serde::Serialize; + +use super::ops::{self, TableParts}; +use super::plan::CheckpointDir; +use super::target::Target; + +/// Status of one compact table. +#[derive(Debug, Clone, Serialize)] +pub struct TargetStatus { + /// Target. + pub target: Target, + /// Whether the materialized view exists. + pub view_exists: bool, + /// Plan shards (0 when no plan). + pub shards_total: usize, + /// Completed shards. + pub shards_done: usize, + /// Whether the gap-closure pass has run. + pub gap_pass_done: bool, + /// Active-part statistics. + pub parts: TableParts, + /// Seconds since the newest `indexed_at` captured in the table. + pub freshness_lag_secs: Option, +} + +/// Collect status for every target. +pub async fn collect( + client: &Client, + database: &str, + checkpoint_dir: &Path, +) -> Result> { + let now = ops::server_now(client).await?; + let mut statuses = Vec::new(); + for target in Target::ALL { + let view_exists = ops::view_created_at(client, database, target.materialized_view()) + .await? + .is_some(); + let checkpoints = CheckpointDir::new(checkpoint_dir, target); + let shards_total = checkpoints.load_plan()?.map_or(0, |p| p.shards.len()); + let shards_done = checkpoints.completed_shards()?.len(); + let gap_pass_done = checkpoints.gap_pass_path().exists(); + + let (parts, freshness_lag_secs) = if view_exists { + let parts = ops::table_parts(client, database, target.table()).await?; + let column = if target == Target::Zaps { + "indexed_at" + } else { + "last_indexed_at" + }; + let newest = client + .query(&format!( + "SELECT toUInt32(max({column})) FROM {database}.{}", + target.table() + )) + .fetch_one::() + .await?; + let lag = (newest > 0).then(|| u64::from(now.saturating_sub(newest))); + (parts, lag) + } else { + (TableParts::default(), None) + }; + + statuses.push(TargetStatus { + target, + view_exists, + shards_total, + shards_done, + gap_pass_done, + parts, + freshness_lag_secs, + }); + } + Ok(statuses) +} + +/// Render statuses as a text table. +pub fn render(statuses: &[TargetStatus]) -> String { + let mut out = String::from( + "target view shards gap parts rows on disk lag\n", + ); + for s in statuses { + out.push_str(&format!( + "{:<12} {:<5} {:<13} {:<4} {:<6} {:<13} {:<13} {}\n", + s.target.name(), + if s.view_exists { "yes" } else { "no" }, + format!("{}/{}", s.shards_done, s.shards_total), + if s.gap_pass_done { "yes" } else { "no" }, + s.parts.parts, + s.parts.rows, + ops::human_bytes(s.parts.bytes_on_disk), + s.freshness_lag_secs + .map_or_else(|| "-".to_string(), |lag| format!("{lag}s")), + )); + } + out +} diff --git a/crates/pensieve-preview/src/lookups/target.rs b/crates/pensieve-preview/src/lookups/target.rs new file mode 100644 index 0000000..3dc1919 --- /dev/null +++ b/crates/pensieve-preview/src/lookups/target.rs @@ -0,0 +1,246 @@ +//! The three compact lookup tables and the SQL that fills them. +//! +//! The backfill `SELECT` bodies here must stay equivalent to the +//! materialized views in `docs/migrations/prepared/017_preview_compact_lookups.sql`. +//! The integration tests insert the same events through both paths and assert +//! identical reads. + +use std::fmt; + +use serde::{Deserialize, Serialize}; + +/// A compact lookup table. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, Serialize, Deserialize, clap::ValueEnum)] +#[serde(rename_all = "snake_case")] +pub enum Target { + /// `profile_latest`: latest kind 0 per pubkey. + Profiles, + /// `zap_receipts_by_target`: one row per (target event, zap receipt). + Zaps, + /// `addressable_latest`: latest event id per (pubkey, kind, d_tag). + Addressable, +} + +/// The kind predicate for replaceable and addressable events. +const ADDRESSABLE_KINDS: &str = + "((kind >= 10000 AND kind < 20000) OR (kind >= 30000 AND kind < 40000))"; + +/// First `d` tag value, identical to the `events_local.d_tag` MATERIALIZED +/// expression. +pub const D_TAG_EXPR: &str = "arrayElement(arrayFilter(t -> t[1] = 'd', tags), 1)[2]"; + +/// Distinct non-empty `e` tag values of an event. +pub const ZAP_TARGETS_EXPR: &str = "arrayDistinct(arrayMap(t -> t[2], \ + arrayFilter(t -> length(t) >= 2 AND t[1] = 'e' AND t[2] != '', tags)))"; + +/// Winner key for replaceable events: newest `created_at`, then lowest id. +const WINNER_KEY_EXPR: &str = "CAST((-toInt64(created_at), id), 'Tuple(Int64, String)')"; + +impl Target { + /// Every target, in rollout order. + pub const ALL: [Self; 3] = [Self::Profiles, Self::Zaps, Self::Addressable]; + + /// Short stable name used in paths and logs. + pub fn name(self) -> &'static str { + match self { + Self::Profiles => "profiles", + Self::Zaps => "zaps", + Self::Addressable => "addressable", + } + } + + /// Target table name. + pub fn table(self) -> &'static str { + match self { + Self::Profiles => "profile_latest", + Self::Zaps => "zap_receipts_by_target", + Self::Addressable => "addressable_latest", + } + } + + /// Materialized view feeding the target from `events_local` inserts. + pub fn materialized_view(self) -> &'static str { + match self { + Self::Profiles => "profile_latest_mv", + Self::Zaps => "zap_receipts_by_target_mv", + Self::Addressable => "addressable_latest_mv", + } + } + + /// Source kind filter on `events_local`. + pub fn kind_predicate(self) -> &'static str { + match self { + Self::Profiles => "kind = 0", + Self::Zaps => "kind = 9735", + Self::Addressable => ADDRESSABLE_KINDS, + } + } + + /// `INSERT ... SELECT` that writes the target rows for source events + /// matching `range_predicate` (a trusted, internally generated SQL + /// predicate such as a `created_at` window). + pub fn backfill_sql(self, database: &str, range_predicate: &str) -> String { + let kinds = self.kind_predicate(); + match self { + Self::Profiles => format!( + "INSERT INTO {database}.profile_latest (pubkey, latest, last_indexed_at) \ + SELECT pubkey, \ + argMinState(CAST((id, content, created_at), 'Tuple(String, String, DateTime)'), \ + {WINNER_KEY_EXPR}) AS latest, \ + max(indexed_at) AS last_indexed_at \ + FROM {database}.events_local \ + WHERE {kinds} AND ({range_predicate}) \ + GROUP BY pubkey" + ), + Self::Zaps => format!( + "INSERT INTO {database}.zap_receipts_by_target \ + (target_event_id, receipt_id, amount_msats, amount_parsed, parser_version, \ + receipt_created_at, indexed_at) \ + WITH \ + (arrayFilter(t -> t[1] = 'bolt11', tags)[1])[2] AS first_bolt11, \ + multiIf( \ + extract(first_bolt11, '^lnbc([0-9]+)m1') != '', \ + toUInt64OrZero(extract(first_bolt11, '^lnbc([0-9]+)m1')) * 100000000, \ + extract(first_bolt11, '^lnbc([0-9]+)u1') != '', \ + toUInt64OrZero(extract(first_bolt11, '^lnbc([0-9]+)u1')) * 100000, \ + extract(first_bolt11, '^lnbc([0-9]+)n1') != '', \ + toUInt64OrZero(extract(first_bolt11, '^lnbc([0-9]+)n1')) * 100, \ + extract(first_bolt11, '^lnbc([0-9]+)p1') != '', \ + intDiv(toUInt64OrZero(extract(first_bolt11, '^lnbc([0-9]+)p1')), 10), \ + toUInt64(0)) AS raw_amount_msats \ + SELECT \ + target_event_id, \ + id AS receipt_id, \ + if(raw_amount_msats > toUInt64(1000000000), toUInt64(0), raw_amount_msats) AS amount_msats, \ + toUInt8(arrayExists(t -> length(t) >= 2 AND t[1] = 'bolt11' \ + AND match(t[2], '^lnbc[0-9]+[munp]1'), tags)) AS amount_parsed, \ + toUInt32(1) AS parser_version, \ + created_at AS receipt_created_at, \ + indexed_at \ + FROM ( \ + SELECT id, created_at, tags, indexed_at \ + FROM {database}.events_local \ + WHERE {kinds} AND ({range_predicate}) \ + ) \ + ARRAY JOIN {ZAP_TARGETS_EXPR} AS target_event_id" + ), + Self::Addressable => format!( + "INSERT INTO {database}.addressable_latest \ + (pubkey, kind, d_tag, latest, last_indexed_at) \ + SELECT pubkey, kind, {D_TAG_EXPR} AS d_tag, \ + argMinState(id, {WINNER_KEY_EXPR}) AS latest, \ + max(indexed_at) AS last_indexed_at \ + FROM {database}.events_local \ + WHERE {kinds} AND ({range_predicate}) \ + GROUP BY pubkey, kind, d_tag" + ), + } + } + + /// Per-hour source row histogram used to plan `created_at` shards. + pub fn hour_histogram_sql(self, database: &str) -> String { + format!( + "SELECT toUInt32(toStartOfHour(created_at)) AS hour, count() AS rows \ + FROM {database}.events_local \ + WHERE {} \ + GROUP BY hour ORDER BY hour", + self.kind_predicate() + ) + } + + /// Source statistics for disk estimation. Returns one row with + /// `source_rows`, `distinct_keys`, `output_rows` and `avg_payload_bytes`. + pub fn source_stats_sql(self, database: &str) -> String { + let kinds = self.kind_predicate(); + match self { + Self::Profiles => format!( + "SELECT count() AS source_rows, uniq(pubkey) AS distinct_keys, \ + distinct_keys AS output_rows, \ + ifNotFinite(avg(length(content)), 0) AS avg_payload_bytes \ + FROM {database}.events_local WHERE {kinds}" + ), + Self::Zaps => format!( + "SELECT count() AS source_rows, uniq(id) AS distinct_keys, \ + sum(length({ZAP_TARGETS_EXPR})) AS output_rows, \ + toFloat64(0) AS avg_payload_bytes \ + FROM {database}.events_local WHERE {kinds}" + ), + Self::Addressable => format!( + "SELECT count() AS source_rows, \ + uniq(pubkey, kind, {D_TAG_EXPR}) AS distinct_keys, \ + distinct_keys AS output_rows, \ + ifNotFinite(avg(length({D_TAG_EXPR})), 0) AS avg_payload_bytes \ + FROM {database}.events_local WHERE {kinds}" + ), + } + } + + /// Estimated stored bytes per output row, before compression, split into + /// (hex/fixed bytes, free-text payload bytes). + pub fn row_bytes(self, avg_payload_bytes: f64) -> (f64, f64) { + match self { + // pubkey + (id, content, created_at) payload + (-created_at, id) key + // + last_indexed_at + length prefixes. + Self::Profiles => ( + 64.0 + 64.0 + 4.0 + 8.0 + 64.0 + 4.0 + 8.0, + avg_payload_bytes, + ), + // target + receipt + amount + parsed + version + 2 timestamps. + Self::Zaps => (64.0 + 64.0 + 8.0 + 1.0 + 4.0 + 4.0 + 4.0 + 4.0, 0.0), + // pubkey + kind + id payload + (-created_at, id) key + last_indexed_at. + Self::Addressable => ( + 64.0 + 2.0 + 64.0 + 8.0 + 64.0 + 4.0 + 8.0, + avg_payload_bytes, + ), + } + } + + /// Whether the target aggregates many source rows into one row per key + /// (so a shard writes at most one state per key). + pub fn aggregates_per_key(self) -> bool { + !matches!(self, Self::Zaps) + } +} + +impl fmt::Display for Target { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + f.write_str(self.name()) + } +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn backfill_sql_is_scoped_and_bind_free() { + for target in Target::ALL { + let sql = target.backfill_sql("scratch", "created_at >= toDateTime(5)"); + assert!(sql.starts_with(&format!("INSERT INTO scratch.{}", target.table()))); + assert!(sql.contains("FROM scratch.events_local")); + assert!(sql.contains(target.kind_predicate())); + assert!(sql.contains("(created_at >= toDateTime(5))")); + assert!(!sql.contains("nostr.")); + // `?` is the clickhouse crate's bind placeholder. + assert!( + !sql.contains('?'), + "{target} backfill SQL must not contain '?'" + ); + } + } + + #[test] + fn source_sql_is_bind_free() { + for target in Target::ALL { + assert!(!target.hour_histogram_sql("db").contains('?')); + assert!(!target.source_stats_sql("db").contains('?')); + } + } + + #[test] + fn names_are_distinct() { + let mut names: Vec<_> = Target::ALL.iter().map(|t| t.table()).collect(); + names.dedup(); + assert_eq!(names.len(), 3); + } +} diff --git a/crates/pensieve-preview/src/lookups/validate.rs b/crates/pensieve-preview/src/lookups/validate.rs new file mode 100644 index 0000000..e26fdce --- /dev/null +++ b/crates/pensieve-preview/src/lookups/validate.rs @@ -0,0 +1,680 @@ +//! Validate compact lookups against the source events and the legacy reads. +//! +//! Keys are sampled deterministically from `events_local` (not from the +//! compact tables, so missing keys are detected): a hashed random subset plus +//! the hottest keys. For each key the harness: +//! +//! 1. reads every source version and computes the expected answer with the +//! independent Rust [`oracle`](super::oracle); +//! 2. skips the key if any source row was indexed after the cutoff, so both +//! sides are compared at a consistent point in time; +//! 3. requires the compact read (the exact function the preview uses) to +//! equal the oracle — any difference is **unexplained**; +//! 4. compares the legacy read and classifies differences as known legacy +//! defects (timestamp tie-breaks, id/content from different rows, +//! duplicate inflation, missing amount rows) or unexplained. + +use std::collections::{BTreeMap, BTreeSet, HashMap}; +use std::time::Duration; + +use anyhow::{Context, Result}; +use clickhouse::Client; +use serde::{Deserialize, Serialize}; + +use super::compact; +use super::ops; +use super::oracle::{self, SourceEvent}; +use super::plan::projection_client; +use super::target::{D_TAG_EXPR, Target, ZAP_TARGETS_EXPR}; +use crate::query; + +/// Validation options. +#[derive(Debug, Clone)] +pub struct ValidateOptions { + /// Targets to validate. + pub targets: Vec, + /// Random keys per target. + pub sample: usize, + /// Hottest keys per target. + pub hot: usize, + /// Sampling seed. + pub seed: u64, + /// Random keys are drawn from `cityHash64(key, seed) % 10000 < permyriad`. + pub sample_permyriad: u32, + /// Compare only data indexed at least this long ago. + pub cutoff_margin: Duration, + /// Keys per source query. + pub batch: usize, + /// Maximum legacy per-key reads for zaps and naddr (each scans a kind slice). + pub legacy_sample: usize, + /// Force sampling and source reads through `events_by_kind`. + pub require_projection: bool, + /// Differences to keep as examples per target. + pub max_examples: usize, +} + +impl Default for ValidateOptions { + fn default() -> Self { + Self { + targets: Target::ALL.to_vec(), + sample: 500, + hot: 50, + seed: 17, + sample_permyriad: 100, + cutoff_margin: Duration::from_secs(300), + batch: 200, + legacy_sample: 100, + require_projection: true, + max_examples: 20, + } + } +} + +/// Classification of one comparison. +#[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord, Serialize)] +#[serde(rename_all = "snake_case")] +pub enum Outcome { + /// Results agree. + Match, + /// Legacy picked a different event among several with the newest + /// `created_at` (the compact rule picks the lowest id). + LegacyTieBreak, + /// Legacy returned an id and content from different events. + LegacyIdContentMix, + /// Legacy counted unmerged duplicate rows. + LegacyDuplicateInflation, + /// Legacy missed receipts lacking a `zap_amounts_data` row. + LegacyAmountRowMissing, + /// Not explained by a known legacy defect. + Unexplained, +} + +/// A recorded difference. +#[derive(Debug, Clone, Serialize)] +pub struct Difference { + /// Which comparison (`compact` vs source, or `legacy` vs source). + pub side: &'static str, + /// Lookup key. + pub key: String, + /// Classification. + pub outcome: Outcome, + /// Human-readable detail. + pub detail: String, +} + +/// Validation results for one target. +#[derive(Debug, Clone, Serialize)] +pub struct TargetReport { + /// Target. + pub target: Target, + /// Keys sampled. + pub sampled: usize, + /// Keys skipped because source rows were indexed after the cutoff. + pub skipped_after_cutoff: usize, + /// Compact-vs-source outcomes. + pub compact: BTreeMap, + /// Legacy-vs-source outcomes. + pub legacy: BTreeMap, + /// Example differences. + pub examples: Vec, +} + +impl TargetReport { + fn new(target: Target, sampled: usize) -> Self { + Self { + target, + sampled, + skipped_after_cutoff: 0, + compact: BTreeMap::new(), + legacy: BTreeMap::new(), + examples: Vec::new(), + } + } + + fn record( + &mut self, + side: &'static str, + key: &str, + outcome: Outcome, + detail: impl FnOnce() -> String, + max_examples: usize, + ) { + let counts = if side == "compact" { + &mut self.compact + } else { + &mut self.legacy + }; + *counts.entry(outcome).or_default() += 1; + if outcome != Outcome::Match && self.examples.len() < max_examples { + self.examples.push(Difference { + side, + key: key.to_string(), + outcome, + detail: detail(), + }); + } + } + + /// Unexplained differences on either side. + pub fn unexplained(&self) -> u64 { + self.compact + .get(&Outcome::Unexplained) + .copied() + .unwrap_or(0) + + self.legacy.get(&Outcome::Unexplained).copied().unwrap_or(0) + } +} + +/// Full validation report. +#[derive(Debug, Clone, Serialize)] +pub struct ValidationReport { + /// Cutoff (Unix seconds): only data indexed at or before this is compared. + pub cutoff: u32, + /// Sampling seed. + pub seed: u64, + /// Per-target results. + pub targets: Vec, +} + +impl ValidationReport { + /// Total unexplained differences. + pub fn unexplained(&self) -> u64 { + self.targets.iter().map(TargetReport::unexplained).sum() + } + + /// Text summary. + pub fn render(&self) -> String { + let mut out = format!("cutoff {} (seed {})\n", self.cutoff, self.seed); + for t in &self.targets { + out.push_str(&format!( + "{}: sampled {}, skipped after cutoff {}\n compact vs source: {:?}\n legacy vs source: {:?}\n", + t.target, t.sampled, t.skipped_after_cutoff, t.compact, t.legacy + )); + for d in &t.examples { + out.push_str(&format!( + " [{}] {:?} {}: {}\n", + d.side, d.outcome, d.key, d.detail + )); + } + } + out.push_str(&format!( + "unexplained differences: {}\n", + self.unexplained() + )); + out + } +} + +#[derive(Debug, Clone, Deserialize, clickhouse::Row)] +struct SourceRow { + id: String, + pubkey: String, + created_at: u32, + kind: u16, + content: String, + tags: Vec>, + indexed_at: u32, +} + +impl SourceRow { + fn event(&self) -> SourceEvent { + SourceEvent { + id: self.id.clone(), + pubkey: self.pubkey.clone(), + created_at: self.created_at, + kind: self.kind, + content: self.content.clone(), + tags: self.tags.clone(), + } + } +} + +/// Run validation. `client` should carry resource limits. +pub async fn run( + client: &Client, + database: &str, + options: &ValidateOptions, +) -> Result { + let now = ops::server_now(client).await?; + let cutoff = now.saturating_sub(options.cutoff_margin.as_secs() as u32); + let source = projection_client(client, options.require_projection); + let ctx = Ctx { + db: database, + source: &source, + compact: client, + legacy: client, + options, + cutoff, + }; + + let mut targets = Vec::new(); + for target in &options.targets { + let report = match target { + Target::Profiles => validate_profiles(&ctx).await?, + Target::Zaps => validate_zaps(&ctx).await?, + Target::Addressable => validate_addressable(&ctx).await?, + }; + targets.push(report); + } + Ok(ValidationReport { + cutoff, + seed: options.seed, + targets, + }) +} + +struct Ctx<'a> { + db: &'a str, + source: &'a Client, + compact: &'a Client, + legacy: &'a Client, + options: &'a ValidateOptions, + cutoff: u32, +} + +fn placeholders(n: usize) -> String { + vec!["?"; n].join(", ") +} + +fn merge_keys(random: Vec, hot: Vec) -> Vec { + let mut seen = BTreeSet::new(); + random + .into_iter() + .chain(hot) + .filter(|k| seen.insert(k.clone())) + .collect() +} + +async fn validate_profiles(ctx: &Ctx<'_>) -> Result { + let (db, o) = (ctx.db, ctx.options); + let random = ctx + .source + .query(&format!( + "SELECT pubkey FROM {db}.events_local \ + WHERE kind = 0 AND cityHash64(pubkey, toUInt64({seed})) % 10000 < {p} \ + GROUP BY pubkey ORDER BY cityHash64(pubkey, toUInt64({seed})) LIMIT {n}", + seed = o.seed, + p = o.sample_permyriad, + n = o.sample + )) + .fetch_all::() + .await + .context("sampling profile keys")?; + let hot = ctx + .source + .query(&format!( + "SELECT pubkey FROM {db}.events_local WHERE kind = 0 \ + GROUP BY pubkey ORDER BY count() DESC, pubkey LIMIT {}", + o.hot + )) + .fetch_all::() + .await + .context("sampling hot profile keys")?; + let keys = merge_keys(random, hot); + let mut report = TargetReport::new(Target::Profiles, keys.len()); + + for batch in keys.chunks(o.batch.max(1)) { + let sql = format!( + "SELECT id, pubkey, toUInt32(created_at) AS created_at, kind, content, tags, \ + toUInt32(indexed_at) AS indexed_at \ + FROM {db}.events_local WHERE kind = 0 AND pubkey IN ({})", + placeholders(batch.len()) + ); + let mut q = ctx.source.query(&sql); + for k in batch { + q = q.bind(k); + } + let rows = q + .fetch_all::() + .await + .context("reading profile sources")?; + let mut by_key: HashMap<&str, Vec<&SourceRow>> = HashMap::new(); + for row in &rows { + by_key.entry(row.pubkey.as_str()).or_default().push(row); + } + + let compact_rows = compact::fetch_profiles(ctx.compact, batch).await?; + let legacy_rows = query::fetch_profiles(ctx.legacy, batch).await?; + + for key in batch { + let versions = by_key.get(key.as_str()).cloned().unwrap_or_default(); + if versions.iter().any(|r| r.indexed_at > ctx.cutoff) { + report.skipped_after_cutoff += 1; + continue; + } + let events: Vec = versions.iter().map(|r| r.event()).collect(); + let expected = oracle::replaceable_winner(&events); + + let compact_outcome = match (expected, compact_rows.get(key)) { + (None, None) => Outcome::Match, + (Some(e), Some(c)) if c.event_id == e.id && c.content == e.content => { + Outcome::Match + } + _ => Outcome::Unexplained, + }; + report.record( + "compact", + key, + compact_outcome, + || { + format!( + "expected {:?}, compact {:?}", + expected.map(|e| &e.id), + compact_rows.get(key).map(|c| &c.event_id) + ) + }, + o.max_examples, + ); + + let legacy = legacy_rows.get(key); + let legacy_outcome = match (expected, legacy) { + (None, None) => Outcome::Match, + (Some(e), Some(l)) if l.event_id == e.id && l.content == e.content => { + Outcome::Match + } + (Some(_), Some(l)) => { + let id_event = events.iter().find(|e| e.id == l.event_id); + if id_event.is_some_and(|e| e.content != l.content) { + Outcome::LegacyIdContentMix + } else if oracle::tie_width(&events) > 1 { + Outcome::LegacyTieBreak + } else { + Outcome::Unexplained + } + } + _ => Outcome::Unexplained, + }; + report.record( + "legacy", + key, + legacy_outcome, + || { + format!( + "expected {:?}, legacy {:?}, tie width {}", + expected.map(|e| &e.id), + legacy.map(|l| &l.event_id), + oracle::tie_width(&events) + ) + }, + o.max_examples, + ); + } + } + Ok(report) +} + +async fn validate_zaps(ctx: &Ctx<'_>) -> Result { + let (db, o) = (ctx.db, ctx.options); + let random = ctx + .source + .query(&format!( + "SELECT target FROM (SELECT tags FROM {db}.events_local WHERE kind = 9735) \ + ARRAY JOIN {ZAP_TARGETS_EXPR} AS target \ + WHERE cityHash64(target, toUInt64({seed})) % 10000 < {p} \ + GROUP BY target ORDER BY cityHash64(target, toUInt64({seed})) LIMIT {n}", + seed = o.seed, + p = o.sample_permyriad, + n = o.sample + )) + .fetch_all::() + .await + .context("sampling zap targets")?; + let hot = ctx + .source + .query(&format!( + "SELECT target FROM (SELECT tags FROM {db}.events_local WHERE kind = 9735) \ + ARRAY JOIN {ZAP_TARGETS_EXPR} AS target \ + GROUP BY target ORDER BY count() DESC, target LIMIT {}", + o.hot + )) + .fetch_all::() + .await + .context("sampling hot zap targets")?; + let keys = merge_keys(random, hot); + let mut report = TargetReport::new(Target::Zaps, keys.len()); + let mut legacy_checked = 0usize; + + for batch in keys.chunks(o.batch.max(1)) { + let sql = format!( + "SELECT id, pubkey, toUInt32(created_at) AS created_at, kind, '' AS content, tags, \ + toUInt32(indexed_at) AS indexed_at \ + FROM {db}.events_local \ + WHERE kind = 9735 AND hasAny({ZAP_TARGETS_EXPR}, [{}])", + placeholders(batch.len()) + ); + let mut q = ctx.source.query(&sql); + for k in batch { + q = q.bind(k); + } + let rows = q + .fetch_all::() + .await + .context("reading zap receipts")?; + let mut by_target: HashMap> = HashMap::new(); + for row in &rows { + for target in oracle::zap_targets(&row.tags) { + by_target.entry(target).or_default().push(row); + } + } + + for key in batch { + let receipts = by_target.get(key).cloned().unwrap_or_default(); + if receipts.iter().any(|r| r.indexed_at > ctx.cutoff) { + report.skipped_after_cutoff += 1; + continue; + } + let events: Vec = receipts.iter().map(|r| r.event()).collect(); + let expected = oracle::zap_totals_by_target(&events) + .remove(key.as_str()) + .unwrap_or_default(); + + let compact_totals = compact::fetch_zap_totals(ctx.compact, key).await?; + let outcome = if compact_totals == expected { + Outcome::Match + } else { + Outcome::Unexplained + }; + report.record( + "compact", + key, + outcome, + || format!("expected {expected:?}, compact {compact_totals:?}"), + o.max_examples, + ); + + if legacy_checked >= o.legacy_sample { + continue; + } + legacy_checked += 1; + let legacy = query::fetch_zap_totals(ctx.legacy, key).await?; + let outcome = if legacy == expected { + Outcome::Match + } else if legacy.zap_count > expected.zap_count && zap_duplicates(ctx, key).await? > 0 { + Outcome::LegacyDuplicateInflation + } else if legacy.zap_count < expected.zap_count { + Outcome::LegacyAmountRowMissing + } else { + Outcome::Unexplained + }; + report.record( + "legacy", + key, + outcome, + || format!("expected {expected:?}, legacy {legacy:?}"), + o.max_examples, + ); + } + } + Ok(report) +} + +/// Unmerged duplicate rows among a target's receipts in `events_local` and +/// `zap_amounts_data`. +async fn zap_duplicates(ctx: &Ctx<'_>, target: &str) -> Result { + let db = ctx.db; + ctx.source + .query(&format!( + "WITH receipts AS ( \ + SELECT id FROM {db}.events_local \ + WHERE kind = 9735 AND has({ZAP_TARGETS_EXPR}, ?) \ + ) \ + SELECT (SELECT count() - uniqExact(id) FROM receipts) \ + + (SELECT count() - uniqExact(event_id) FROM {db}.zap_amounts_data \ + WHERE event_id IN (SELECT id FROM receipts))" + )) + .bind(target) + .fetch_one::() + .await + .context("counting zap duplicates") +} + +async fn validate_addressable(ctx: &Ctx<'_>) -> Result { + let (db, o) = (ctx.db, ctx.options); + let kinds = Target::Addressable.kind_predicate(); + let random = ctx + .source + .query(&format!( + "SELECT pubkey, kind, {D_TAG_EXPR} AS d FROM {db}.events_local \ + WHERE {kinds} AND cityHash64(pubkey, kind, d, toUInt64({seed})) % 10000 < {p} \ + GROUP BY pubkey, kind, d \ + ORDER BY cityHash64(pubkey, kind, d, toUInt64({seed})) LIMIT {n}", + seed = o.seed, + p = o.sample_permyriad, + n = o.sample + )) + .fetch_all::<(String, u16, String)>() + .await + .context("sampling naddr coordinates")?; + let hot = ctx + .source + .query(&format!( + "SELECT pubkey, kind, {D_TAG_EXPR} AS d FROM {db}.events_local WHERE {kinds} \ + GROUP BY pubkey, kind, d ORDER BY count() DESC, pubkey, kind, d LIMIT {}", + o.hot + )) + .fetch_all::<(String, u16, String)>() + .await + .context("sampling hot naddr coordinates")?; + let keys = merge_keys(random, hot); + let mut report = TargetReport::new(Target::Addressable, keys.len()); + let mut legacy_checked = 0usize; + + for batch in keys.chunks(o.batch.max(1)) { + let pubkeys: BTreeSet<&str> = batch.iter().map(|(p, _, _)| p.as_str()).collect(); + let kind_list: BTreeSet = batch.iter().map(|(_, k, _)| *k).collect(); + let kind_list = kind_list + .iter() + .map(u16::to_string) + .collect::>() + .join(", "); + let sql = format!( + "SELECT id, pubkey, toUInt32(created_at) AS created_at, kind, '' AS content, tags, \ + toUInt32(indexed_at) AS indexed_at \ + FROM {db}.events_local \ + WHERE {kinds} AND kind IN ({kind_list}) AND pubkey IN ({})", + placeholders(pubkeys.len()) + ); + let mut q = ctx.source.query(&sql); + for p in &pubkeys { + q = q.bind(*p); + } + let rows = q + .fetch_all::() + .await + .context("reading naddr sources")?; + let mut by_key: HashMap<(&str, u16, &str), Vec<&SourceRow>> = HashMap::new(); + for row in &rows { + by_key + .entry((row.pubkey.as_str(), row.kind, oracle::d_tag(&row.tags))) + .or_default() + .push(row); + } + + for (pubkey, kind, d) in batch { + let label = format!("{kind}:{pubkey}:{d}"); + let versions = by_key + .get(&(pubkey.as_str(), *kind, d.as_str())) + .cloned() + .unwrap_or_default(); + if versions.iter().any(|r| r.indexed_at > ctx.cutoff) { + report.skipped_after_cutoff += 1; + continue; + } + let events: Vec = versions.iter().map(|r| r.event()).collect(); + let expected = oracle::replaceable_winner(&events).map(|e| e.id.clone()); + + let compact_id = compact::fetch_replaceable(ctx.compact, *kind, pubkey, d) + .await? + .map(|e| e.id); + let outcome = if compact_id == expected { + Outcome::Match + } else { + Outcome::Unexplained + }; + report.record( + "compact", + &label, + outcome, + || format!("expected {expected:?}, compact {compact_id:?}"), + o.max_examples, + ); + + if legacy_checked >= o.legacy_sample { + continue; + } + legacy_checked += 1; + let legacy_id = query::fetch_replaceable(ctx.legacy, *kind, pubkey, d) + .await? + .map(|e| e.id); + let outcome = if legacy_id == expected { + Outcome::Match + } else if oracle::tie_width(&events) > 1 { + Outcome::LegacyTieBreak + } else { + Outcome::Unexplained + }; + report.record( + "legacy", + &label, + outcome, + || { + format!( + "expected {expected:?}, legacy {legacy_id:?}, tie width {}", + oracle::tie_width(&events) + ) + }, + o.max_examples, + ); + } + } + Ok(report) +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn merge_keys_dedupes_and_keeps_order() { + let merged = merge_keys(vec!["b", "a"], vec!["a", "c", "b"]); + assert_eq!(merged, vec!["b", "a", "c"]); + } + + #[test] + fn report_counts_unexplained_on_both_sides() { + let mut report = TargetReport::new(Target::Profiles, 3); + report.record("compact", "k1", Outcome::Match, String::new, 5); + report.record("compact", "k2", Outcome::Unexplained, || "x".into(), 5); + report.record("legacy", "k1", Outcome::LegacyTieBreak, || "y".into(), 5); + report.record("legacy", "k2", Outcome::Unexplained, || "z".into(), 1); + assert_eq!(report.unexplained(), 2); + // The last record's cap of 1 is already reached, so it is not kept. + assert_eq!(report.examples.len(), 2); + } + + #[test] + fn placeholders_format() { + assert_eq!(placeholders(3), "?, ?, ?"); + assert_eq!(placeholders(1), "?"); + } +} diff --git a/crates/pensieve-preview/src/query.rs b/crates/pensieve-preview/src/query.rs index fc4c966..4e39b9c 100644 --- a/crates/pensieve-preview/src/query.rs +++ b/crates/pensieve-preview/src/query.rs @@ -1,7 +1,11 @@ //! ClickHouse query layer for fetching events, profiles, and engagement data. //! -//! All queries are point lookups by primary key or small result sets, -//! designed for sub-10ms response times. +//! These are the legacy reads against `events_local`. Profile, zap and naddr +//! reads filter by kind and therefore rely on the `events_by_kind` projection +//! (or the pubkey bloom filter) for speed. The compact replacements live in +//! [`crate::lookups::compact`] and are selected per lookup by +//! [`crate::lookups::Lookups`]. These functions stay available for comparison +//! and rollback while the projection remains intact. use clickhouse::Client; use serde::Deserialize; @@ -103,6 +107,15 @@ pub struct EngagementCounts { pub zap_count: u64, } +/// Zap totals for a single target event. +#[derive(Debug, Clone, Copy, Default, PartialEq, Eq, Deserialize, clickhouse::Row)] +pub struct ZapTotals { + /// Total zap amount in millisatoshis. + pub total_msats: u64, + /// Number of zap receipts. + pub zap_count: u64, +} + /// Fetch a single event by its hex ID. pub async fn fetch_event_by_id( client: &Client, @@ -204,10 +217,20 @@ pub async fn fetch_replaceable( Ok(result) } -/// Fetch engagement counts (reactions, comments, reposts) for an event. +/// Fetch engagement counts (reactions, comments, reposts, zaps) for an event. pub async fn fetch_engagement( client: &Client, event_id: &str, +) -> Result { + fetch_engagement_with_zaps(client, event_id, fetch_zap_totals(client, event_id)).await +} + +/// Fetch reaction, comment and repost counts in parallel with a caller-chosen +/// zap totals query. +pub(crate) async fn fetch_engagement_with_zaps( + client: &Client, + event_id: &str, + zaps: impl std::future::Future>, ) -> Result { // Fetch all three counts in parallel using separate queries // (ClickHouse handles these as fast point lookups on SummingMergeTree tables) @@ -217,12 +240,6 @@ pub async fn fetch_engagement( count: u64, } - #[derive(Debug, Deserialize, clickhouse::Row)] - struct ZapRow { - total_msats: u64, - zap_count: u64, - } - let (reactions, comments, reposts, zaps) = tokio::try_join!( async { client @@ -263,26 +280,7 @@ pub async fn fetch_engagement( .map(|r| r.map_or(0, |r| r.count)) .map_err(PreviewError::from) }, - async { - client - .query( - "SELECT sum(z.amount_msats) AS total_msats, count() AS zap_count \ - FROM events_local e \ - INNER JOIN zap_amounts_data z ON e.id = z.event_id \ - WHERE e.kind = 9735 \ - AND arrayExists(t -> t[1] = 'e' AND t[2] = ?, e.tags)", - ) - .bind(event_id) - .fetch_optional::() - .await - .map(|r| { - r.unwrap_or(ZapRow { - total_msats: 0, - zap_count: 0, - }) - }) - .map_err(PreviewError::from) - }, + zaps, )?; Ok(EngagementCounts { @@ -294,6 +292,25 @@ pub async fn fetch_engagement( }) } +/// Fetch zap totals for an event by scanning kind 9735 receipts in +/// `events_local` (legacy path; relies on the `events_by_kind` projection). +pub async fn fetch_zap_totals(client: &Client, event_id: &str) -> Result { + let totals = client + .query( + "SELECT sum(z.amount_msats) AS total_msats, count() AS zap_count \ + FROM events_local e \ + INNER JOIN zap_amounts_data z ON e.id = z.event_id \ + WHERE e.kind = 9735 \ + AND arrayExists(t -> t[1] = 'e' AND t[2] = ?, e.tags)", + ) + .bind(event_id) + .fetch_optional::() + .await? + .unwrap_or_default(); + + Ok(totals) +} + /// Batch-fetch display names for a set of pubkeys. /// /// Returns a map of pubkey (hex) -> display name. Pubkeys without @@ -329,20 +346,28 @@ pub async fn fetch_display_names( let rows = query.fetch_all::().await?; - let names = rows + Ok(display_names_from_profiles( + rows.into_iter().map(|row| (row.pubkey, row.content)), + )) +} + +/// Map `(pubkey, kind 0 content)` pairs to display names, dropping profiles +/// without a usable name. +pub(crate) fn display_names_from_profiles( + profiles: impl IntoIterator, +) -> std::collections::HashMap { + profiles .into_iter() - .filter_map(|row| { - let meta = ProfileMetadata::from_json(&row.content); + .filter_map(|(pubkey, content)| { + let meta = ProfileMetadata::from_json(&content); let name = meta.display_name().to_string(); if name == "Anonymous" { None } else { - Some((row.pubkey, name)) + Some((pubkey, name)) } }) - .collect(); - - Ok(names) + .collect() } #[cfg(test)] diff --git a/crates/pensieve-preview/src/render/mod.rs b/crates/pensieve-preview/src/render/mod.rs index aedc256..4aa5595 100644 --- a/crates/pensieve-preview/src/render/mod.rs +++ b/crates/pensieve-preview/src/render/mod.rs @@ -19,7 +19,7 @@ use std::collections::HashMap; use maud::Markup; -use crate::query::{self, ProfileMetadata}; +use crate::query::ProfileMetadata; use crate::resolve::ResolvedContent; use crate::state::AppState; @@ -65,7 +65,10 @@ pub async fn render_page( let display_names = if referenced_pubkeys.is_empty() { HashMap::new() } else { - query::fetch_display_names(&state.clickhouse, &referenced_pubkeys).await? + state + .lookups + .fetch_display_names(&referenced_pubkeys) + .await? }; // Fetch quoted events for inline card rendering (depth 1 only) @@ -121,25 +124,24 @@ pub async fn render_page( .find(|t| t.len() >= 2 && t[0] == "e") .map(|t| t[1].clone()); - let (original_event, original_author, original_npub) = if let Some(ref id) = - reposted_id - { - let orig = query::fetch_event_by_id(&state.clickhouse, id).await?; - if let Some(ref orig_event) = orig { - let orig_profile = - query::fetch_profile(&state.clickhouse, &orig_event.pubkey).await?; - let orig_meta = - orig_profile.map(|p| ProfileMetadata::from_json(&p.content)); - let orig_npub = nostr::PublicKey::from_hex(&orig_event.pubkey) - .ok() - .and_then(|pk| nostr::ToBech32::to_bech32(&pk).ok()); - (Some(orig_event.clone()), orig_meta, orig_npub) + let (original_event, original_author, original_npub) = + if let Some(ref id) = reposted_id { + let orig = state.lookups.fetch_event_by_id(id).await?; + if let Some(ref orig_event) = orig { + let orig_profile = + state.lookups.fetch_profile(&orig_event.pubkey).await?; + let orig_meta = + orig_profile.map(|p| ProfileMetadata::from_json(&p.content)); + let orig_npub = nostr::PublicKey::from_hex(&orig_event.pubkey) + .ok() + .and_then(|pk| nostr::ToBech32::to_bech32(&pk).ok()); + (Some(orig_event.clone()), orig_meta, orig_npub) + } else { + (None, None, None) + } } else { (None, None, None) - } - } else { - (None, None, None) - }; + }; Ok(repost::render( event, @@ -228,7 +230,7 @@ async fn fetch_quoted_events( let futures: Vec<_> = event_ids .iter() .map(|id| async { - let row = query::fetch_event_by_id(&state.clickhouse, id).await; + let row = state.lookups.fetch_event_by_id(id).await; (id.clone(), row) }) .collect(); @@ -247,7 +249,7 @@ async fn fetch_quoted_events( let profile_futures: Vec<_> = found .iter() .map(|(id, row)| async { - let profile = query::fetch_profile(&state.clickhouse, &row.pubkey).await; + let profile = state.lookups.fetch_profile(&row.pubkey).await; (id.clone(), profile) }) .collect(); diff --git a/crates/pensieve-preview/src/resolve.rs b/crates/pensieve-preview/src/resolve.rs index 078972c..c372b21 100644 --- a/crates/pensieve-preview/src/resolve.rs +++ b/crates/pensieve-preview/src/resolve.rs @@ -7,7 +7,7 @@ use nostr::nips::nip19::{FromBech32, Nip19, Nip19Event, Nip19Profile}; use nostr::{EventId, PublicKey, ToBech32}; use crate::error::PreviewError; -use crate::query::{self, EngagementCounts, EventRow, ProfileMetadata}; +use crate::query::{EngagementCounts, EventRow, ProfileMetadata}; use crate::state::AppState; /// The resolved result of a NIP-19 identifier lookup. @@ -81,7 +81,7 @@ pub async fn resolve(state: &AppState, identifier: &str) -> Result ( @@ -136,7 +136,9 @@ async fn resolve_event_by_id( ) -> Result { let id_hex = event_id.to_hex(); - let event = query::fetch_event_by_id(&state.clickhouse, &id_hex) + let event = state + .lookups + .fetch_event_by_id(&id_hex) .await? .ok_or_else(|| PreviewError::NotFound(format!("event {id_hex}")))?; @@ -150,7 +152,9 @@ async fn resolve_replaceable( pubkey: &str, d_tag: &str, ) -> Result { - let event = query::fetch_replaceable(&state.clickhouse, kind, pubkey, d_tag) + let event = state + .lookups + .fetch_replaceable(kind, pubkey, d_tag) .await? .ok_or_else(|| { PreviewError::NotFound(format!("replaceable event {kind}:{pubkey}:{d_tag}")) @@ -175,13 +179,14 @@ async fn build_event_content( // Fetch author profile, engagement, and reply parent name in parallel let (author_profile, engagement, reply_author_name) = tokio::try_join!( - query::fetch_profile(&state.clickhouse, &author_pubkey_hex), - query::fetch_engagement(&state.clickhouse, &event_id_hex), + state.lookups.fetch_profile(&author_pubkey_hex), + state.lookups.fetch_engagement(&event_id_hex), async { if let Some(parent_pk) = &reply_parent_pubkey { - let names = - query::fetch_display_names(&state.clickhouse, std::slice::from_ref(parent_pk)) - .await?; + let names = state + .lookups + .fetch_display_names(std::slice::from_ref(parent_pk)) + .await?; Ok(names.get(parent_pk).cloned()) } else { Ok(None) diff --git a/crates/pensieve-preview/src/routes/json.rs b/crates/pensieve-preview/src/routes/json.rs index 4145fcd..04a061a 100644 --- a/crates/pensieve-preview/src/routes/json.rs +++ b/crates/pensieve-preview/src/routes/json.rs @@ -17,7 +17,7 @@ use axum::http::{HeaderMap, HeaderValue, StatusCode, header}; use axum::response::{IntoResponse, Response}; use crate::error::PreviewError; -use crate::query::{self, ProfileMetadata}; +use crate::query::ProfileMetadata; use crate::resolve; use crate::state::AppState; @@ -84,7 +84,9 @@ pub async fn json_handler_inner( mentioned_pubkeys.dedup(); mentioned_pubkeys.truncate(MAX_MENTIONS); - let profiles = query::fetch_profiles(&state.clickhouse, &mentioned_pubkeys) + let profiles = state + .lookups + .fetch_profiles(&mentioned_pubkeys) .await .unwrap_or_default(); let mut mentions_map = serde_json::Map::new(); diff --git a/crates/pensieve-preview/src/state.rs b/crates/pensieve-preview/src/state.rs index 2cd2962..ba2c3d7 100644 --- a/crates/pensieve-preview/src/state.rs +++ b/crates/pensieve-preview/src/state.rs @@ -6,6 +6,7 @@ use clickhouse::Client; use moka::future::Cache; use crate::config::Config; +use crate::lookups::Lookups; /// Cached HTML response with metadata for TTL decisions. #[derive(Clone, Debug)] @@ -42,6 +43,9 @@ pub struct AppState { /// ClickHouse client for database queries. pub clickhouse: Client, + /// Preview read layer (legacy or compact source per lookup). + pub lookups: Lookups, + /// Application configuration. pub config: Arc, @@ -77,8 +81,11 @@ impl AppState { "application state initialized" ); + let lookups = Lookups::new(clickhouse.clone(), config.compact_lookups); + Self { clickhouse, + lookups, config: Arc::new(config), cache, og_cache, diff --git a/crates/pensieve-preview/tests/compact_lookups.rs b/crates/pensieve-preview/tests/compact_lookups.rs new file mode 100644 index 0000000..1b508af --- /dev/null +++ b/crates/pensieve-preview/tests/compact_lookups.rs @@ -0,0 +1,1146 @@ +//! Integration tests for the compact preview lookups against a real +//! ClickHouse (24.8, matching production). +//! +//! Run with a disposable server: +//! +//! ```text +//! just ch-test-up +//! just test-clickhouse +//! ``` +//! +//! Each test creates its own scratch database containing a production-shaped +//! `events_local` (with the `events_by_kind` projection), the legacy +//! `zap_amounts_data` objects (migrations 003 + 009) and migration 017. +//! Assertions run twice: with merges stopped (so duplicate rows and states +//! are physically present) and after `OPTIMIZE ... FINAL`. + +use std::collections::HashMap; +use std::sync::atomic::{AtomicU64, Ordering}; +use std::time::Duration; + +use clickhouse::Client; +use serde::Serialize; + +use pensieve_preview::lookups::backfill::{self, BackfillOptions, StopReason}; +use pensieve_preview::lookups::oracle::{self, SourceEvent}; +use pensieve_preview::lookups::plan::{self, CheckpointDir, PlanOptions, ShardBy}; +use pensieve_preview::lookups::target::Target; +use pensieve_preview::lookups::{CompactLookups, Lookups, bench, compact, validate}; +use pensieve_preview::query::{self, ZapTotals}; + +const IGNORE_REASON: &str = "requires a disposable ClickHouse at PENSIEVE_TEST_CLICKHOUSE_URL"; +const BASE_SCHEMA: &str = include_str!("fixtures/base_schema.sql"); +const MIGRATION_017: &str = + include_str!("../../../docs/migrations/prepared/017_preview_compact_lookups.sql"); + +// --------------------------------------------------------------------------- +// Harness +// --------------------------------------------------------------------------- + +struct TestDb { + name: String, + admin: Client, + client: Client, +} + +impl TestDb { + async fn new() -> Self { + let url = std::env::var("PENSIEVE_TEST_CLICKHOUSE_URL") + .unwrap_or_else(|_| panic!("{IGNORE_REASON}")); + static COUNTER: AtomicU64 = AtomicU64::new(0); + let nanos = std::time::SystemTime::now() + .duration_since(std::time::UNIX_EPOCH) + .unwrap() + .as_nanos(); + let name = format!( + "pv_test_{nanos}_{}", + COUNTER.fetch_add(1, Ordering::Relaxed) + ); + let admin = Client::default().with_url(&url); + admin + .query(&format!("CREATE DATABASE {name}")) + .execute() + .await + .expect("create database"); + let client = Client::default().with_url(&url).with_database(&name); + let db = Self { + name, + admin, + client, + }; + db.apply(BASE_SCHEMA).await; + db + } + + async fn apply(&self, sql: &str) { + for statement in statements(sql, &self.name) { + self.admin + .query(&statement) + .execute() + .await + .unwrap_or_else(|e| panic!("{e}\n{statement}")); + } + } + + async fn apply_migration(&self) { + self.apply(MIGRATION_017).await; + } + + async fn exec(&self, sql: &str) { + self.admin + .query(sql) + .execute() + .await + .unwrap_or_else(|e| panic!("{e}\n{sql}")); + } + + async fn stop_merges(&self) { + for table in [ + "events_local", + "zap_amounts_data", + "profile_latest", + "zap_receipts_by_target", + "addressable_latest", + ] { + self.exec(&format!("SYSTEM STOP MERGES {}.{table}", self.name)) + .await; + } + } + + async fn merge_all(&self) { + for table in [ + "events_local", + "zap_amounts_data", + "profile_latest", + "zap_receipts_by_target", + "addressable_latest", + ] { + self.exec(&format!("SYSTEM START MERGES {}.{table}", self.name)) + .await; + self.exec(&format!("OPTIMIZE TABLE {}.{table} FINAL", self.name)) + .await; + } + } + + /// Insert with production async-insert settings (fire and forget), then + /// flush the queue so the rows are visible. + async fn insert_async(&self, events: &[SourceEvent]) { + let client = self + .client + .clone() + .with_option("async_insert", "1") + .with_option("wait_for_async_insert", "0"); + insert_rows(&client, events).await; + self.exec("SYSTEM FLUSH ASYNC INSERT QUEUE").await; + } + + /// Insert synchronously; with merges stopped each call is its own part. + async fn insert_sync(&self, events: &[SourceEvent]) { + let client = self.client.clone().with_option("async_insert", "0"); + insert_rows(&client, events).await; + } + + async fn count(&self, sql: &str) -> u64 { + self.client + .query(sql) + .fetch_one::() + .await + .unwrap_or_else(|e| panic!("{e}\n{sql}")) + } + + async fn drop(self) { + self.admin + .query(&format!("DROP DATABASE IF EXISTS {}", self.name)) + .execute() + .await + .expect("drop database"); + } +} + +/// Split a SQL file into statements, qualify `nostr.` with the scratch +/// database and drop comments. +fn statements(sql: &str, database: &str) -> Vec { + let mut out = Vec::new(); + let mut current = String::new(); + for line in sql.lines() { + let code = match line.find("--") { + Some(i) => &line[..i], + None => line, + }; + if code.trim().is_empty() { + continue; + } + current.push_str(code); + current.push('\n'); + if code.trim_end().ends_with(';') { + let statement = current.trim().trim_end_matches(';').to_string(); + out.push(statement.replace("nostr.", &format!("{database}."))); + current.clear(); + } + } + assert!( + current.trim().is_empty(), + "unterminated statement: {current}" + ); + for statement in &out { + assert!( + !statement.contains('?'), + "statement contains '?': {statement}" + ); + } + out +} + +#[derive(Serialize, clickhouse::Row)] +struct InsertRow<'a> { + id: &'a str, + pubkey: &'a str, + created_at: u32, + kind: u16, + content: &'a str, + sig: &'a str, + tags: &'a [Vec], + relay_source: &'a str, +} + +async fn insert_rows(client: &Client, events: &[SourceEvent]) { + let mut insert = client.insert("events_local").expect("insert"); + for e in events { + insert + .write(&InsertRow { + id: &e.id, + pubkey: &e.pubkey, + created_at: e.created_at, + kind: e.kind, + content: &e.content, + sig: "", + tags: &e.tags, + relay_source: "", + }) + .await + .expect("write row"); + } + insert.end().await.expect("end insert"); +} + +fn hex(n: u64) -> String { + format!("{n:064x}") +} + +fn tags(raw: &[&[&str]]) -> Vec> { + raw.iter() + .map(|t| t.iter().map(|s| s.to_string()).collect()) + .collect() +} + +fn event( + id: u64, + pubkey: u64, + created_at: u32, + kind: u16, + content: &str, + t: &[&[&str]], +) -> SourceEvent { + SourceEvent { + id: hex(id), + pubkey: hex(pubkey), + created_at, + kind, + content: content.to_string(), + tags: tags(t), + } +} + +fn profile(id: u64, pubkey: u64, created_at: u32, name: &str) -> SourceEvent { + event( + id, + pubkey, + created_at, + 0, + &format!("{{\"name\":\"{name}\"}}"), + &[], + ) +} + +fn receipt(id: u64, t: &[&[&str]]) -> SourceEvent { + event(id, 0xbeef, 1_700_000_000, 9735, "", t) +} + +fn backfill_options() -> BackfillOptions { + BackfillOptions { + delay_between_shards: Duration::ZERO, + grace: Duration::ZERO, + min_free_bytes: 0, + ..BackfillOptions::default() + } +} + +/// Assert every compact lookup matches the oracle over `events`. +async fn assert_matches_oracle(db: &TestDb, events: &[SourceEvent], phase: &str) { + let c = &db.client; + + let profiles = oracle::latest_profiles(events); + let pubkeys: Vec = profiles.keys().cloned().collect(); + let batch = compact::fetch_profiles(c, &pubkeys).await.unwrap(); + assert_eq!(batch.len(), profiles.len(), "{phase}: profile batch size"); + for (pubkey, expected) in &profiles { + let single = compact::fetch_profile(c, pubkey).await.unwrap().unwrap(); + assert_eq!( + single.event_id, expected.id, + "{phase}: profile id for {pubkey}" + ); + assert_eq!(single.content, expected.content, "{phase}: profile content"); + assert_eq!( + batch[pubkey].event_id, expected.id, + "{phase}: batch profile id" + ); + assert_eq!( + batch[pubkey].content, expected.content, + "{phase}: batch content" + ); + } + + let zaps = oracle::zap_totals_by_target(events); + for (target, expected) in &zaps { + let got = compact::fetch_zap_totals(c, target).await.unwrap(); + assert_eq!(&got, expected, "{phase}: zap totals for {target}"); + } + + let addressable = oracle::latest_addressable(events); + for ((pubkey, kind, d), expected) in &addressable { + let got = compact::fetch_replaceable(c, *kind, pubkey, d) + .await + .unwrap() + .unwrap_or_else(|| panic!("{phase}: missing {kind}:{pubkey}:{d}")); + assert_eq!(got.id, expected.id, "{phase}: naddr {kind}:{pubkey}:{d}"); + } +} + +// --------------------------------------------------------------------------- +// Schema +// --------------------------------------------------------------------------- + +#[tokio::test] +#[ignore = "requires a disposable ClickHouse at PENSIEVE_TEST_CLICKHOUSE_URL"] +async fn migration_creates_pinned_schema_and_is_idempotent() { + let db = TestDb::new().await; + db.apply_migration().await; + // Re-applying is a no-op. + db.apply_migration().await; + + let columns = db + .client + .query( + "SELECT table, name, type FROM system.columns WHERE database = currentDatabase() \ + AND table IN ('profile_latest', 'zap_receipts_by_target', 'addressable_latest') \ + ORDER BY table, position", + ) + .fetch_all::<(String, String, String)>() + .await + .unwrap(); + let get = |table: &str, name: &str| { + columns + .iter() + .find(|(t, n, _)| t == table && n == name) + .map(|(_, _, ty)| ty.clone()) + .unwrap_or_else(|| panic!("missing {table}.{name}")) + }; + assert_eq!( + get("profile_latest", "latest"), + "AggregateFunction(argMin, Tuple(String, String, DateTime), Tuple(Int64, String))" + ); + assert_eq!( + get("addressable_latest", "latest"), + "AggregateFunction(argMin, String, Tuple(Int64, String))" + ); + assert_eq!(get("zap_receipts_by_target", "parser_version"), "UInt32"); + + let views = db + .count( + "SELECT count() FROM system.tables WHERE database = currentDatabase() \ + AND engine = 'MaterializedView' AND name LIKE '%latest_mv' OR name = 'zap_receipts_by_target_mv'", + ) + .await; + assert!(views >= 3); + db.drop().await; +} + +// --------------------------------------------------------------------------- +// Profiles +// --------------------------------------------------------------------------- + +#[tokio::test] +#[ignore = "requires a disposable ClickHouse at PENSIEVE_TEST_CLICKHOUSE_URL"] +async fn profiles_handle_duplicates_out_of_order_and_ties() { + let db = TestDb::new().await; + db.apply_migration().await; + db.stop_merges().await; + + let newer = profile(0x50, 1, 200, "newer"); + let older = profile(0x10, 1, 100, "older"); + let tie_high = profile(0x90, 2, 300, "tie-high-id"); + let tie_low = profile(0x30, 2, 300, "tie-low-id"); + let tie2_low = profile(0x31, 3, 300, "tie2-low-id"); + let tie2_high = profile(0x91, 3, 300, "tie2-high-id"); + + // Out of order: newest first. Ties inserted in both orders. Replays. + let inserts = [ + vec![newer.clone()], + vec![older.clone()], + vec![newer.clone()], + vec![tie_high.clone()], + vec![tie_low.clone()], + vec![tie2_low.clone()], + vec![tie2_high.clone()], + vec![newer.clone(), older.clone()], + ]; + for batch in &inserts { + db.insert_sync(batch).await; + } + let all: Vec = inserts.concat(); + + // Duplicate states are physically present before merges. + let raw_states = db.count("SELECT count() FROM profile_latest").await; + assert!( + raw_states > 3, + "expected unmerged duplicate states, got {raw_states}" + ); + + for phase in ["before merges", "after merges"] { + let c = &db.client; + let p1 = compact::fetch_profile(c, &hex(1)).await.unwrap().unwrap(); + assert_eq!( + (p1.event_id.as_str(), p1.content.as_str()), + (newer.id.as_str(), newer.content.as_str()), + "{phase}" + ); + let p2 = compact::fetch_profile(c, &hex(2)).await.unwrap().unwrap(); + assert_eq!(p2.event_id, tie_low.id, "{phase}: lowest id wins a tie"); + assert_eq!( + p2.content, tie_low.content, + "{phase}: content from the same event" + ); + let p3 = compact::fetch_profile(c, &hex(3)).await.unwrap().unwrap(); + assert_eq!(p3.event_id, tie2_low.id, "{phase}: tie inserted low-first"); + assert!(compact::fetch_profile(c, &hex(4)).await.unwrap().is_none()); + + let batch = compact::fetch_profiles(c, &[hex(1), hex(2), hex(3), hex(4)]) + .await + .unwrap(); + assert_eq!(batch.len(), 3, "{phase}: missing pubkeys are omitted"); + assert_eq!(batch[&hex(2)].event_id, tie_low.id); + + let names = compact::fetch_display_names(c, &[hex(1), hex(2), hex(4)]) + .await + .unwrap(); + assert_eq!(names[&hex(1)], "newer"); + assert_eq!(names[&hex(2)], "tie-low-id"); + assert!(!names.contains_key(&hex(4))); + + assert_matches_oracle(&db, &all, phase).await; + db.merge_all().await; + } + + assert_eq!(db.count("SELECT count() FROM profile_latest").await, 3); + db.drop().await; +} + +// --------------------------------------------------------------------------- +// Zaps +// --------------------------------------------------------------------------- + +#[tokio::test] +#[ignore = "requires a disposable ClickHouse at PENSIEVE_TEST_CLICKHOUSE_URL"] +async fn zaps_dedupe_replays_and_match_legacy_amount_rules() { + let db = TestDb::new().await; + db.apply_migration().await; + db.stop_merges().await; + + let t1 = hex(0x7001); + let t2 = hex(0x7002); + let receipts = vec![ + // 21 sats, duplicate e tag. + receipt( + 1, + &[ + &["bolt11", "lnbc210n1abc"], + &["e", &t1], + &["e", &t1], + &["p", "x"], + ], + ), + // 500 sats to two targets. + receipt(2, &[&["bolt11", "lnbc5u1abc"], &["e", &t1], &["e", &t2]]), + // Unparseable invoice: not counted. + receipt(3, &[&["bolt11", "lnbc100x"], &["e", &t1]]), + // Above 1M sats: counted with amount 0. + receipt(4, &[&["bolt11", "lnbc11m1abc"], &["e", &t1]]), + // First bolt11 invalid, second valid: counted, amount 0. + receipt( + 5, + &[&["bolt11", "nope"], &["bolt11", "lnbc5u1abc"], &["e", &t2]], + ), + // Pico amount. + receipt(6, &[&["bolt11", "lnbc12345p1abc"], &["e", &t2]]), + // No targets at all. + receipt(7, &[&["bolt11", "lnbc5u1abc"], &["e"], &["e", ""]]), + ]; + // Original inserts, then a full replay and a partial replay in separate parts. + db.insert_sync(&receipts).await; + db.insert_sync(&receipts).await; + db.insert_sync(&receipts[..2]).await; + let mut all = receipts.clone(); + all.extend_from_slice(&receipts); + all.extend_from_slice(&receipts[..2]); + + let raw_t1 = db + .count(&format!( + "SELECT count() FROM zap_receipts_by_target WHERE target_event_id = '{t1}'" + )) + .await; + assert!(raw_t1 > 4, "expected unmerged replayed rows, got {raw_t1}"); + + let expected = oracle::zap_totals_by_target(&all); + assert_eq!( + expected[&t1], + ZapTotals { + total_msats: 21_000 + 500_000, + zap_count: 3 + } + ); + assert_eq!( + expected[&t2], + ZapTotals { + total_msats: 500_000 + 1_234, + zap_count: 3 + } + ); + + // Legacy double counts replays before merges. + let legacy_before = query::fetch_zap_totals(&db.client, &t1).await.unwrap(); + assert!(legacy_before.zap_count > expected[&t1].zap_count); + + for phase in ["before merges", "after merges"] { + assert_matches_oracle(&db, &all, phase).await; + let none = compact::fetch_zap_totals(&db.client, &hex(0x7003)) + .await + .unwrap(); + assert_eq!(none, ZapTotals::default()); + db.merge_all().await; + } + + // Once legacy inputs are merged, legacy and compact agree exactly: + // same inclusion and amount rules as migration 009. + for target in [&t1, &t2] { + let legacy = query::fetch_zap_totals(&db.client, target).await.unwrap(); + let compact = compact::fetch_zap_totals(&db.client, target).await.unwrap(); + assert_eq!(legacy, compact, "legacy parity for {target}"); + } + db.drop().await; +} + +#[tokio::test] +#[ignore = "requires a disposable ClickHouse at PENSIEVE_TEST_CLICKHOUSE_URL"] +async fn zap_corrections_win_regardless_of_arrival_order() { + let db = TestDb::new().await; + db.apply_migration().await; + db.stop_merges().await; + let target = hex(0x8001); + + // A corrected parser (version 2) writes rows directly. Receipt 1's v2 row + // arrives after its v1 row; receipt 2's v2 row arrives before the receipt. + let r1 = receipt(1, &[&["bolt11", "lnbc210n1abc"], &["e", &target]]); + let r2 = receipt(2, &[&["bolt11", "lnbc5u1abc"], &["e", &target]]); + let correction = |receipt_id: &str, amount: u64, parsed: u8| { + format!( + "INSERT INTO {}.zap_receipts_by_target VALUES \ + ('{target}', '{receipt_id}', {amount}, {parsed}, 2, toDateTime(1700000000), now())", + db.name + ) + }; + + db.insert_sync(std::slice::from_ref(&r1)).await; + assert_eq!( + compact::fetch_zap_totals(&db.client, &target) + .await + .unwrap(), + ZapTotals { + total_msats: 21_000, + zap_count: 1 + } + ); + db.exec(&correction(&r1.id, 42_000, 1)).await; + db.exec(&correction(&r2.id, 0, 0)).await; + db.insert_sync(std::slice::from_ref(&r2)).await; + // A late amount change in the legacy table has no effect on compact rows. + db.exec(&format!( + "INSERT INTO {}.zap_amounts_data VALUES ('{}', '', toDateTime(1800000000), 999999, '', '')", + db.name, r1.id + )) + .await; + + for phase in ["before merges", "after merges"] { + let got = compact::fetch_zap_totals(&db.client, &target) + .await + .unwrap(); + assert_eq!( + got, + ZapTotals { + total_msats: 42_000, + zap_count: 1 + }, + "{phase}: v2 rows win; v2 marks receipt 2 unparsed" + ); + db.merge_all().await; + } + db.drop().await; +} + +// --------------------------------------------------------------------------- +// naddr +// --------------------------------------------------------------------------- + +#[tokio::test] +#[ignore = "requires a disposable ClickHouse at PENSIEVE_TEST_CLICKHOUSE_URL"] +async fn addressable_latest_handles_ties_and_falls_back_for_uncovered_kinds() { + let db = TestDb::new().await; + db.apply_migration().await; + db.stop_merges().await; + + let author = 0xa11ce; + let events = vec![ + event( + 0x40, + author, + 20, + 30023, + "v2-high", + &[&["d", "slug"], &["title", "T"]], + ), + event(0x10, author, 10, 30023, "v1", &[&["d", "slug"]]), + event(0x20, author, 20, 30023, "v2-low", &[&["d", "slug"]]), + event(0x30, author, 5, 30023, "other", &[&["d", "other"]]), + event(0x50, author, 7, 10002, "relays", &[&["r", "wss://x"]]), + event(0x60, author, 9, 30023, "no-d", &[&["d"]]), + // Kind 1 is not covered by addressable_latest. + event(0x70, author, 9, 1, "note", &[&["d", "n"]]), + ]; + for e in &events { + db.insert_sync(std::slice::from_ref(e)).await; + } + db.insert_sync(&events).await; + + for phase in ["before merges", "after merges"] { + let c = &db.client; + let pk = hex(author); + let slug = compact::fetch_replaceable(c, 30023, &pk, "slug") + .await + .unwrap() + .unwrap(); + assert_eq!(slug.id, hex(0x20), "{phase}: lowest id among newest"); + assert_eq!(slug.content, "v2-low"); + assert_eq!(slug.d_tag, "slug"); + let other = compact::fetch_replaceable(c, 30023, &pk, "other") + .await + .unwrap() + .unwrap(); + assert_eq!(other.id, hex(0x30)); + let relays = compact::fetch_replaceable(c, 10002, &pk, "") + .await + .unwrap() + .unwrap(); + assert_eq!(relays.id, hex(0x50)); + let no_d = compact::fetch_replaceable(c, 30023, &pk, "") + .await + .unwrap() + .unwrap(); + assert_eq!(no_d.id, hex(0x60)); + assert!( + compact::fetch_replaceable(c, 30023, &pk, "missing") + .await + .unwrap() + .is_none() + ); + + // The dispatcher uses the legacy query for uncovered kinds. + let lookups = Lookups::new(c.clone(), CompactLookups::ALL); + let note = lookups + .fetch_replaceable(1, &pk, "n") + .await + .unwrap() + .unwrap(); + assert_eq!(note.id, hex(0x70)); + let via_dispatch = lookups + .fetch_replaceable(30023, &pk, "slug") + .await + .unwrap() + .unwrap(); + assert_eq!(via_dispatch.id, hex(0x20)); + + assert_matches_oracle(&db, &events, phase).await; + db.merge_all().await; + } + db.drop().await; +} + +// --------------------------------------------------------------------------- +// Ingest safety +// --------------------------------------------------------------------------- + +#[tokio::test] +#[ignore = "requires a disposable ClickHouse at PENSIEVE_TEST_CLICKHOUSE_URL"] +async fn async_insert_flush_survives_malformed_events() { + let db = TestDb::new().await; + db.apply_migration().await; + + let malformed: Vec = vec![ + event(1, 1, 0, 0, "", &[]), + event(2, 1, u32::MAX, 0, "not json \u{0}", &[&[]]), + event( + 3, + 2, + 5, + 9735, + "", + &[&[], &[""], &["e"], &["e", ""], &["bolt11"]], + ), + event( + 4, + 2, + 5, + 9735, + "", + &[ + &["bolt11", "lnbc99999999999999999999999999m1x"], + &["e", "t"], + ], + ), + event( + 5, + 2, + 5, + 9735, + "", + &[&["bolt11", "lnbc"], &["bolt11", "LNBC10U1"], &["e", "t"]], + ), + event( + 6, + 2, + 5, + 9735, + "", + &[&["bolt11", "lnbc18446744073709551615u1x"], &["e", "t"]], + ), + event(7, 3, 5, 30023, "", &[&["d"]]), + event(8, 3, 5, 30023, "", &[]), + event(9, 3, 5, 10002, "", &[&["d", "\u{1F600}"]]), + event(10, 3, 5, 39999, "", &[&["d", ""], &["d", "second"]]), + ]; + db.insert_async(&malformed).await; + db.exec("SYSTEM FLUSH LOGS").await; + + assert_eq!( + db.count("SELECT uniqExact(id) FROM events_local").await, + malformed.len() as u64, + "every row must land despite malformed tags" + ); + let failed = db + .count(&format!( + "SELECT count() FROM system.asynchronous_insert_log \ + WHERE database = '{}' AND status != 'Ok'", + db.name + )) + .await; + assert_eq!(failed, 0, "no failed async-insert flushes"); + assert_matches_oracle(&db, &malformed, "malformed").await; + db.drop().await; +} + +// --------------------------------------------------------------------------- +// Backfill +// --------------------------------------------------------------------------- + +/// A deterministic mixed workload: profiles with ties, receipts with +/// replays, addressable versions. +fn workload(seed: u64, created_base: u32) -> Vec { + let mut events = Vec::new(); + for i in 0..40u64 { + let pubkey = 0x1000 + i % 9; + let at = created_base + ((i / 2) as u32) * 3600; + events.push(profile( + seed * 1000 + i, + pubkey, + at, + &format!("p{seed}-{i}"), + )); + let target = hex(0x2000 + i % 5); + let invoice = format!("lnbc{}n1x", (i % 7) + 1); + events.push(event( + seed * 1000 + 500 + i, + 0xbeef, + at, + 9735, + "", + &[&["bolt11", &invoice], &["e", &target]], + )); + let d = format!("d{}", i % 4); + events.push(event( + seed * 1000 + 800 + i, + 0x3000 + i % 3, + at, + 30023, + "", + &[&["d", &d]], + )); + } + events +} + +#[tokio::test] +#[ignore = "requires a disposable ClickHouse at PENSIEVE_TEST_CLICKHOUSE_URL"] +async fn backfill_and_live_ingest_overlap_without_gaps_or_double_counts() { + let db = TestDb::new().await; + let checkpoint_dir = tempfile::tempdir().unwrap(); + + // A: history that predates the views. + let a = workload(1, 1_600_000_000); + db.insert_sync(&a).await; + + db.apply_migration().await; + db.stop_merges().await; + + // B: live traffic after the views exist, including replays of history and + // events whose created_at falls inside shards the backfill will cover. + let mut b = workload(2, 1_600_000_000); + b.extend_from_slice(&a[..30]); + db.insert_async(&b).await; + + let options = PlanOptions { + shard_by: ShardBy::CreatedAt, + max_rows_per_shard: 10, + id_prefix_len: 2, + require_projection: false, + }; + let mut plans = Vec::new(); + for target in Target::ALL { + let p = plan::build_plan(&db.client, &db.name, target, &options) + .await + .unwrap(); + assert!(p.shards.len() > 2, "{target}: expected several shards"); + let dir = CheckpointDir::new(checkpoint_dir.path(), target); + dir.save_plan(&p, false).unwrap(); + plans.push((p, dir)); + } + + // Run one shard at a time, inserting more live traffic (C) between shards: + // new events, replays, and events landing in already-completed shards. + let mut c_events = Vec::new(); + let one_shard = BackfillOptions { + max_shards: Some(1), + ..backfill_options() + }; + let mut round = 0u64; + loop { + let mut complete = true; + for (p, dir) in &plans { + let report = backfill::run_backfill(&db.client, p, dir, &one_shard) + .await + .unwrap(); + if report.stopped != StopReason::Complete { + complete = false; + } + } + round += 1; + let mut live = workload(10 + round, 1_600_000_000 + (round as u32 % 3) * 7200); + live.truncate(9); + live.extend_from_slice(&a[(round as usize % 20)..(round as usize % 20) + 3]); + db.insert_async(&live).await; + c_events.extend(live); + if complete { + break; + } + assert!(round < 200, "backfill did not finish"); + } + + let mut all = a.clone(); + all.extend(b.clone()); + all.extend(c_events.clone()); + assert_matches_oracle(&db, &all, "after interleaved backfill (before merges)").await; + + // Re-running is a no-op; a full second backfill (replan) is idempotent; + // the gap-closure pass is harmless. + for (p, dir) in &plans { + let again = backfill::run_backfill(&db.client, p, dir, &backfill_options()) + .await + .unwrap(); + assert_eq!(again.shards_run, 0); + assert_eq!(again.stopped, StopReason::Complete); + dir.save_plan(p, true).unwrap(); + let full = backfill::run_backfill(&db.client, p, dir, &backfill_options()) + .await + .unwrap(); + assert_eq!(full.shards_run, p.shards.len()); + backfill::run_gap_pass( + &db.client, + p, + dir, + Duration::from_secs(3600), + &backfill_options(), + ) + .await + .unwrap(); + } + assert_matches_oracle(&db, &all, "after repeated backfill (before merges)").await; + + db.merge_all().await; + assert_matches_oracle(&db, &all, "after merges").await; + db.drop().await; +} + +#[tokio::test] +#[ignore = "requires a disposable ClickHouse at PENSIEVE_TEST_CLICKHOUSE_URL"] +async fn backfill_and_materialized_views_produce_identical_rows() { + let db = TestDb::new().await; + let checkpoint_dir = tempfile::tempdir().unwrap(); + let mut events = workload(3, 1_650_000_000); + events.extend(workload(4, 1_650_000_000)); + events.push(receipt( + 9_001, + &[ + &["bolt11", "nope"], + &["bolt11", "lnbc7u1x"], + &["e", &hex(0x2001)], + ], + )); + events.push(event( + 9_002, + 0x3001, + 1_650_000_000, + 30023, + "", + &[&["d"], &["d", "x"]], + )); + + // Path 1: backfill only. + db.insert_sync(&events).await; + db.apply_migration().await; + for target in Target::ALL { + let options = PlanOptions { + shard_by: ShardBy::CreatedAt, + max_rows_per_shard: 25, + id_prefix_len: 2, + require_projection: false, + }; + let p = plan::build_plan(&db.client, &db.name, target, &options) + .await + .unwrap(); + let dir = CheckpointDir::new(checkpoint_dir.path(), target); + dir.save_plan(&p, false).unwrap(); + let report = backfill::run_backfill(&db.client, &p, &dir, &backfill_options()) + .await + .unwrap(); + assert_eq!(report.stopped, StopReason::Complete); + } + assert_matches_oracle(&db, &events, "backfill path").await; + let snapshot = |db: &TestDb| { + let client = db.client.clone(); + async move { + let profiles = client + .query("SELECT pubkey, toString(argMinMerge(latest)) FROM profile_latest GROUP BY pubkey ORDER BY pubkey") + .fetch_all::<(String, String)>() + .await + .unwrap(); + let zaps = client + .query( + "SELECT target_event_id, receipt_id, argMax(amount_msats, parser_version), \ + argMax(amount_parsed, parser_version) FROM zap_receipts_by_target \ + GROUP BY target_event_id, receipt_id ORDER BY 1, 2", + ) + .fetch_all::<(String, String, u64, u8)>() + .await + .unwrap(); + let addressable = client + .query( + "SELECT pubkey, kind, d_tag, argMinMerge(latest) FROM addressable_latest \ + GROUP BY pubkey, kind, d_tag ORDER BY 1, 2, 3", + ) + .fetch_all::<(String, u16, String, String)>() + .await + .unwrap(); + (profiles, zaps, addressable) + } + }; + let via_backfill = snapshot(&db).await; + + // Path 2: the same events through the materialized views. + for target in Target::ALL { + db.exec(&format!("TRUNCATE TABLE {}.{}", db.name, target.table())) + .await; + } + db.insert_async(&events).await; + let via_views = snapshot(&db).await; + + assert_eq!(via_backfill, via_views); + db.drop().await; +} + +/// Bulk synthetic history large enough for the optimizer to choose +/// `events_by_kind`, as in production. +async fn insert_bulk(db: &TestDb, rows: u64) { + db.exec(&format!( + "INSERT INTO {}.events_local (id, pubkey, created_at, kind, content, sig, tags) \ + SELECT lower(hex(SHA256(toString(number)))) AS id, \ + lower(hex(SHA256(toString(number % 3000)))) AS pubkey, \ + toDateTime(1600000000 + intDiv(number, 6000) * 600) AS created_at, \ + [0, 0, 1, 1, 7, 3, 9735, 9735, 30023, 10002, 6, 1][number % 12 + 1] AS kind, \ + concat('{{\"name\":\"u', toString(number), '\"}}') AS content, \ + '' AS sig, \ + multiIf( \ + kind = 9735, [['e', lower(hex(SHA256(toString(number % 400))))], \ + ['bolt11', concat('lnbc', toString(number % 900 + 1), 'n1x')]], \ + kind = 30023, [['d', concat('slug', toString(number % 7))]], \ + [['e', lower(hex(SHA256(toString(number % 777))))]]) AS tags \ + FROM numbers({rows}) SETTINGS async_insert = 0", + db.name + )) + .await; +} + +#[tokio::test] +#[ignore = "requires a disposable ClickHouse at PENSIEVE_TEST_CLICKHOUSE_URL"] +async fn projection_backfill_validate_and_bench_end_to_end() { + let db = TestDb::new().await; + let checkpoint_dir = tempfile::tempdir().unwrap(); + insert_bulk(&db, 600_000).await; + db.apply_migration().await; + + let options = PlanOptions { + shard_by: ShardBy::CreatedAt, + max_rows_per_shard: 20_000, + id_prefix_len: 2, + require_projection: true, + }; + for target in Target::ALL { + let p = plan::build_plan(&db.client, &db.name, target, &options) + .await + .unwrap(); + assert!(p.shards.len() > 1); + assert!(p.estimate.peak_bytes >= p.estimate.pre_merge_bytes); + let dir = CheckpointDir::new(checkpoint_dir.path(), target); + dir.save_plan(&p, false).unwrap(); + + // Interrupted run, then resume. + let first = backfill::run_backfill( + &db.client, + &p, + &dir, + &BackfillOptions { + max_shards: Some(1), + ..backfill_options() + }, + ) + .await + .unwrap(); + assert_eq!(first.stopped, StopReason::MaxShards); + let rest = backfill::run_backfill(&db.client, &p, &dir, &backfill_options()) + .await + .unwrap(); + assert_eq!(rest.stopped, StopReason::Complete); + + let projection = format!("{}.events_local.events_by_kind", db.name); + for checkpoint in dir.completed_shards().unwrap() { + let log = checkpoint.query_log.expect("query_log entry"); + assert!( + log.projections.contains(&projection), + "{target}: {:?}", + log.projections + ); + } + } + + // Profiles: compact equals an independent SQL reference over all rows. + let mismatches = db + .count( + "SELECT count() FROM ( \ + SELECT pubkey, argMin((id, content), (-toInt64(created_at), id)) AS r \ + FROM events_local WHERE kind = 0 GROUP BY pubkey \ + ) AS ref \ + FULL OUTER JOIN ( \ + SELECT pubkey, argMinMerge(latest) AS w FROM profile_latest GROUP BY pubkey \ + ) AS c USING pubkey \ + WHERE tupleElement(ref.r, 1) != tupleElement(c.w, 1) \ + OR tupleElement(ref.r, 2) != tupleElement(c.w, 2)", + ) + .await; + assert_eq!(mismatches, 0); + + let report = validate::run( + &db.client, + &db.name, + &validate::ValidateOptions { + sample: 40, + hot: 5, + sample_permyriad: 2_000, + cutoff_margin: Duration::ZERO, + batch: 25, + legacy_sample: 10, + ..validate::ValidateOptions::default() + }, + ) + .await + .unwrap(); + assert_eq!(report.unexplained(), 0, "{}", report.render()); + for t in &report.targets { + assert!(t.sampled > 0, "{}: no keys sampled", t.target); + } + // The synthetic data has created_at ties per pubkey, which legacy breaks + // arbitrarily. + let profiles = &report.targets[0]; + assert!( + profiles + .compact + .get(&validate::Outcome::Match) + .copied() + .unwrap_or(0) + > 0 + ); + + let bench = bench::run( + &db.client, + &db.name, + &bench::BenchOptions { + samples: 3, + batch_size: 5, + include_legacy: true, + legacy_samples: 1, + ..bench::BenchOptions::default() + }, + ) + .await + .unwrap(); + assert!( + bench.measurements.iter().all(|m| m.error.is_none()), + "{}", + bench.render() + ); + assert!(bench.measurements.iter().any(|m| m.server_ms.is_some())); + assert!( + bench + .storage + .projections + .iter() + .any(|p| p.name == "events_by_kind") + ); + let lookups: HashMap<&str, usize> = + bench.summaries.iter().map(|s| (s.lookup, s.runs)).collect(); + for name in [ + "profile", + "profiles_batch", + "display_names", + "zaps", + "naddr", + ] { + assert!(lookups.contains_key(name), "bench missing {name}"); + } + db.drop().await; +} diff --git a/crates/pensieve-preview/tests/fixtures/base_schema.sql b/crates/pensieve-preview/tests/fixtures/base_schema.sql new file mode 100644 index 0000000..0f6ed54 --- /dev/null +++ b/crates/pensieve-preview/tests/fixtures/base_schema.sql @@ -0,0 +1,101 @@ +-- Test fixture: production-shaped events_local plus the legacy zap_amounts objects +-- (docs/clickhouse_self_hosted.sql, 003_zap_amounts.sql, 009_fix_zap_amounts.sql). +-- Integration tests rewrite the `nostr.` qualifier to a scratch database. + +CREATE TABLE IF NOT EXISTS nostr.events_local ( + -- Event fields (NIP-01) + id String, -- 32-byte hex event ID (SHA-256 hash, always 64 chars) + pubkey String, -- 32-byte hex public key (always 64 chars) + created_at DateTime, -- Unix timestamp when event was created + kind UInt16, -- Event kind (0-65535, see NIP-01) + content String CODEC(ZSTD(3)), -- Event content (arbitrary string) + sig String, -- 64-byte hex Schnorr signature (always 128 chars) + tags Array(Array(String)), -- Nested array of tags + + -- Metadata fields + indexed_at DateTime DEFAULT now(), + relay_source String DEFAULT '', + + -- Materialized columns for video events (computed at insert time) + d_tag String MATERIALIZED arrayElement(arrayFilter(t -> t[1] = 'd', tags), 1)[2], + title String MATERIALIZED arrayElement(arrayFilter(t -> t[1] = 'title', tags), 1)[2], + thumbnail String MATERIALIZED arrayElement(arrayFilter(t -> t[1] = 'thumb', tags), 1)[2], + video_url String MATERIALIZED arrayElement(arrayFilter(t -> t[1] = 'url', tags), 1)[2], + + -- Secondary indexes (fallback for queries not matching projections) + INDEX idx_created_at created_at TYPE minmax GRANULARITY 4, + INDEX idx_kind kind TYPE minmax GRANULARITY 4, + INDEX idx_pubkey pubkey TYPE bloom_filter(0.01) GRANULARITY 4, + + -- Projections for alternate query patterns + -- ClickHouse automatically selects the best projection for each query + PROJECTION events_by_kind ( + SELECT * + ORDER BY (kind, created_at, pubkey) + ), + PROJECTION events_by_author ( + SELECT * + ORDER BY (pubkey, created_at, kind) + ) + +) ENGINE = ReplacingMergeTree(indexed_at) +ORDER BY (id) +SETTINGS index_granularity = 8192, + deduplicate_merge_projection_mode = 'rebuild'; + +CREATE TABLE IF NOT EXISTS nostr.zap_amounts_data ( + event_id String, + zapper_pubkey String, -- pubkey of the zap receipt event (LNURL server) + created_at DateTime, + amount_msats UInt64, -- amount in millisatoshis + recipient_pubkey String, -- p tag: who received the zap + sender_pubkey String -- P tag: who sent the zap (optional, may be empty) +) ENGINE = ReplacingMergeTree(created_at) +ORDER BY (event_id); + +CREATE MATERIALIZED VIEW IF NOT EXISTS nostr.zap_amounts_mv TO nostr.zap_amounts_data +AS SELECT + id AS event_id, + pubkey AS zapper_pubkey, + created_at, + -- Parse bolt11 amount with strict multiplier validation AND sanity threshold + -- bolt11 format: lnbc1 + -- Only accept m, u, n, p multipliers. No multiplier = invalid (not BTC!) + -- Amounts over 1M sats (1B msats) are set to 0 (excluded as fake) + if( + multiIf( + -- milli-BTC: 1m = 100,000 sats = 100,000,000 msats + extract((arrayFilter(t -> t[1] = 'bolt11', tags)[1])[2], '^lnbc([0-9]+)m1') != '', + toUInt64OrZero(extract((arrayFilter(t -> t[1] = 'bolt11', tags)[1])[2], '^lnbc([0-9]+)m1')) * 100000000, + -- micro-BTC: 1u = 100 sats = 100,000 msats + extract((arrayFilter(t -> t[1] = 'bolt11', tags)[1])[2], '^lnbc([0-9]+)u1') != '', + toUInt64OrZero(extract((arrayFilter(t -> t[1] = 'bolt11', tags)[1])[2], '^lnbc([0-9]+)u1')) * 100000, + -- nano-BTC: 1n = 0.1 sats = 100 msats + extract((arrayFilter(t -> t[1] = 'bolt11', tags)[1])[2], '^lnbc([0-9]+)n1') != '', + toUInt64OrZero(extract((arrayFilter(t -> t[1] = 'bolt11', tags)[1])[2], '^lnbc([0-9]+)n1')) * 100, + -- pico-BTC: 1p = 0.0001 sats = 0.1 msats + extract((arrayFilter(t -> t[1] = 'bolt11', tags)[1])[2], '^lnbc([0-9]+)p1') != '', + intDiv(toUInt64OrZero(extract((arrayFilter(t -> t[1] = 'bolt11', tags)[1])[2], '^lnbc([0-9]+)p1')), 10), + -- No valid multiplier = invalid, return 0 + toUInt64(0) + ) > toUInt64(1000000000), -- If over 1M sats (1B msats = 0.01 BTC) + toUInt64(0), -- Set to 0 (exclude as fake) + multiIf( + extract((arrayFilter(t -> t[1] = 'bolt11', tags)[1])[2], '^lnbc([0-9]+)m1') != '', + toUInt64OrZero(extract((arrayFilter(t -> t[1] = 'bolt11', tags)[1])[2], '^lnbc([0-9]+)m1')) * 100000000, + extract((arrayFilter(t -> t[1] = 'bolt11', tags)[1])[2], '^lnbc([0-9]+)u1') != '', + toUInt64OrZero(extract((arrayFilter(t -> t[1] = 'bolt11', tags)[1])[2], '^lnbc([0-9]+)u1')) * 100000, + extract((arrayFilter(t -> t[1] = 'bolt11', tags)[1])[2], '^lnbc([0-9]+)n1') != '', + toUInt64OrZero(extract((arrayFilter(t -> t[1] = 'bolt11', tags)[1])[2], '^lnbc([0-9]+)n1')) * 100, + extract((arrayFilter(t -> t[1] = 'bolt11', tags)[1])[2], '^lnbc([0-9]+)p1') != '', + intDiv(toUInt64OrZero(extract((arrayFilter(t -> t[1] = 'bolt11', tags)[1])[2], '^lnbc([0-9]+)p1')), 10), + toUInt64(0) + ) + ) AS amount_msats, + -- Extract recipient (p tag) + (arrayFilter(t -> t[1] = 'p', tags)[1])[2] AS recipient_pubkey, + -- Extract sender (P tag, optional - may be empty if anonymous zap) + (arrayFilter(t -> t[1] = 'P', tags)[1])[2] AS sender_pubkey +FROM nostr.events_local +WHERE kind = 9735 + AND arrayExists(t -> length(t) >= 2 AND t[1] = 'bolt11' AND match(t[2], '^lnbc[0-9]+[munp]1'), tags); diff --git a/docs/clickhouse_self_hosted.sql b/docs/clickhouse_self_hosted.sql index 03b8b30..8ac76e0 100644 --- a/docs/clickhouse_self_hosted.sql +++ b/docs/clickhouse_self_hosted.sql @@ -105,6 +105,11 @@ ORDER BY (id) SETTINGS index_granularity = 8192, deduplicate_merge_projection_mode = 'rebuild'; +-- Compact preview lookups (profile_latest, zap_receipts_by_target, +-- addressable_latest) are created by +-- docs/migrations/prepared/017_preview_compact_lookups.sql, not here; apply it +-- after this file on fresh installs. See docs/preview_compact_lookups.md. + -- ============================================================================= -- TAG MATERIALIZED VIEW -- ============================================================================= diff --git a/docs/migrations/README.md b/docs/migrations/README.md index 052e53b..e0dc4b8 100644 --- a/docs/migrations/README.md +++ b/docs/migrations/README.md @@ -44,6 +44,7 @@ docker exec -i pensieve-clickhouse clickhouse-client --database nostr < docs/mig | 014_relay_distribution.sql | Pre-aggregate NIP-65 relay distribution data | **Yes** (runs automatically) | | 015_kinds_stats.sql | Pre-aggregate event kind statistics | **Yes** (runs automatically) | | prepared/016_drop_events_by_time_projection.sql | Prepared removal of the full-row time projection; excluded from `ch-migrate-all` | No (background removal mutation) | +| prepared/017_preview_compact_lookups.sql | Compact profile, zap and naddr lookup tables for `pensieve-preview`; excluded from `ch-migrate-all` | **Yes** (`preview-lookups backfill`, bounded and resumable) | > **⚠️ IMPORTANT:** Migration 005 MUST be run if you applied migration 004. The views in 004 do > expensive JOINs at query time that cause 100% CPU usage on large datasets (100M+ events). @@ -465,6 +466,36 @@ the disk reserve, stop and investigate rather than forcing a final optimize. --- +### prepared/017_preview_compact_lookups.sql + +**Purpose:** Let nstr.to's profile, zap and naddr previews read three compact +tables instead of filtering `events_local` by kind through the full-row +`events_by_kind` projection. + +It creates three tables, each fed by an insert-triggered view on +`events_local`: +- `profile_latest`: latest kind 0 per pubkey; +- `zap_receipts_by_target`: one row per (target event, zap receipt), using + the migration 009 amount rules; +- `addressable_latest`: latest event id per (pubkey, kind, d_tag) for kinds + 10000–19999 and 30000–39999. + +Reads stay correct before merges, with replays, and with out-of-order +arrivals. The migration does **not** backfill or touch the projection. + +**Apply, backfill, validate and roll out:** follow +[`ops/RUNBOOK.md` → Preview compact lookups](../../ops/RUNBOOK.md#preview-compact-lookups). +Design and behaviour differences: +[`docs/preview_compact_lookups.md`](../preview_compact_lookups.md). + +**Rollback:** set `PREVIEW_COMPACT_LOOKUPS=none`, restart `pensieve-preview`, then: + +```bash +just ch-migrate docs/migrations/rollback/017_drop_preview_compact_lookups.sql +``` + +--- + ## Full Deployment Checklist When deploying new API endpoints that depend on these migrations: diff --git a/docs/migrations/prepared/017_preview_compact_lookups.sql b/docs/migrations/prepared/017_preview_compact_lookups.sql new file mode 100644 index 0000000..17e1b07 --- /dev/null +++ b/docs/migrations/prepared/017_preview_compact_lookups.sql @@ -0,0 +1,165 @@ +-- Prepared migration: 017_preview_compact_lookups +-- Description: Compact lookup tables for pensieve-preview (nstr.to) so profile, +-- zap and naddr previews no longer depend on the full-row +-- events_by_kind projection on events_local. +-- Date: 2026-10-10 +-- +-- Prepared for ClickHouse 24.8.14.39. +-- +-- To run: +-- just ch-migrate docs/migrations/prepared/017_preview_compact_lookups.sql +-- +-- Rollback: +-- just ch-migrate docs/migrations/rollback/017_drop_preview_compact_lookups.sql +-- +-- IMPORTANT: +-- - This file is deliberately outside docs/migrations/*.sql, so +-- `just ch-migrate-all` cannot apply it. It attaches new insert-triggered +-- materialized views to events_local, which run inside every ingest flush. +-- - It creates empty tables and starts capturing new inserts only. History is +-- loaded separately with the bounded `preview-lookups backfill` tool, which +-- must run while the events_by_kind projection still exists. +-- - Follow docs/preview_compact_lookups.md and the ops/RUNBOOK.md section +-- "Preview compact lookups" for preflight, backfill, validation and rollout. +-- - Every expression below must be non-throwing for arbitrary tag arrays. A +-- throwing materialized view fails the whole async-insert flush, and with +-- wait_for_async_insert=0 the ingester never sees that error. +-- +-- Correctness model (see docs/preview_compact_lookups.md): +-- Reads never depend on background merges. Every read collapses duplicate +-- rows/states with GROUP BY / -Merge, so results are identical before and +-- after merges, and replays, out-of-order arrivals and backfill overlap are +-- all harmless. + +-- ============================================================================= +-- PROFILE LOOKUP: latest kind 0 per pubkey +-- ============================================================================= +-- +-- Winner rule (NIP-01 replaceable): highest created_at wins; on a created_at +-- tie the lowest event id wins. argMin over the key (-created_at, id) encodes +-- exactly that rule, and the payload tuple keeps id, content and created_at +-- from the same event. + +CREATE TABLE IF NOT EXISTS nostr.profile_latest ( + pubkey String, + latest AggregateFunction(argMin, Tuple(String, String, DateTime), Tuple(Int64, String)), + last_indexed_at SimpleAggregateFunction(max, DateTime) +) ENGINE = AggregatingMergeTree +ORDER BY pubkey; + +CREATE MATERIALIZED VIEW IF NOT EXISTS nostr.profile_latest_mv TO nostr.profile_latest +AS SELECT + pubkey, + argMinState( + CAST((id, content, created_at), 'Tuple(String, String, DateTime)'), + CAST((-toInt64(created_at), id), 'Tuple(Int64, String)') + ) AS latest, + max(indexed_at) AS last_indexed_at +FROM nostr.events_local +WHERE kind = 0 +GROUP BY pubkey; + +-- ============================================================================= +-- ZAP LOOKUP: one row per (target event, zap receipt) +-- ============================================================================= +-- +-- The amount is derived from the receipt's own bolt11 tag using exactly the +-- migration 009 rules (amount from the first bolt11 tag; values above 1B msats +-- become 0; included only if some bolt11 tag matches ^lnbc[0-9]+[munp]1). It +-- never joins another table, so there is no ordering dependency between a +-- receipt and its amount. +-- +-- Targets are every distinct non-empty `e` tag value, matching the legacy +-- preview query's arrayExists(t -> t[1] = 'e' AND t[2] = ?) semantics. +-- +-- parser_version allows a corrected parser to run beside this one (as a new +-- materialized view writing version 2) and be backfilled without a gap. Reads +-- take argMax(..., parser_version) per receipt, so the newest parser wins +-- regardless of which version's row arrived first. + +CREATE TABLE IF NOT EXISTS nostr.zap_receipts_by_target ( + target_event_id String, + receipt_id String, + amount_msats UInt64, + amount_parsed UInt8, + parser_version UInt32, + receipt_created_at DateTime, + indexed_at DateTime +) ENGINE = ReplacingMergeTree(parser_version) +ORDER BY (target_event_id, receipt_id); + +CREATE MATERIALIZED VIEW IF NOT EXISTS nostr.zap_receipts_by_target_mv TO nostr.zap_receipts_by_target +AS WITH + (arrayFilter(t -> t[1] = 'bolt11', tags)[1])[2] AS first_bolt11, + multiIf( + extract(first_bolt11, '^lnbc([0-9]+)m1') != '', + toUInt64OrZero(extract(first_bolt11, '^lnbc([0-9]+)m1')) * 100000000, + extract(first_bolt11, '^lnbc([0-9]+)u1') != '', + toUInt64OrZero(extract(first_bolt11, '^lnbc([0-9]+)u1')) * 100000, + extract(first_bolt11, '^lnbc([0-9]+)n1') != '', + toUInt64OrZero(extract(first_bolt11, '^lnbc([0-9]+)n1')) * 100, + extract(first_bolt11, '^lnbc([0-9]+)p1') != '', + intDiv(toUInt64OrZero(extract(first_bolt11, '^lnbc([0-9]+)p1')), 10), + toUInt64(0) + ) AS raw_amount_msats +SELECT + target_event_id, + id AS receipt_id, + if(raw_amount_msats > toUInt64(1000000000), toUInt64(0), raw_amount_msats) AS amount_msats, + toUInt8(arrayExists( + t -> length(t) >= 2 AND t[1] = 'bolt11' AND match(t[2], '^lnbc[0-9]+[munp]1'), + tags + )) AS amount_parsed, + toUInt32(1) AS parser_version, + created_at AS receipt_created_at, + indexed_at +FROM nostr.events_local +ARRAY JOIN arrayDistinct(arrayMap( + t -> t[2], + arrayFilter(t -> length(t) >= 2 AND t[1] = 'e' AND t[2] != '', tags) +)) AS target_event_id +WHERE kind = 9735; + +-- ============================================================================= +-- NADDR LOOKUP: latest event pointer per (pubkey, kind, d_tag) +-- ============================================================================= +-- +-- Covers replaceable (10000-19999) and addressable (30000-39999) kinds. Stores +-- only the winning event id; previews fetch the row from events_local by +-- primary key. d_tag uses the same expression as the events_local MATERIALIZED +-- d_tag column so lookups match the legacy `d_tag = ?` filter exactly. Same +-- winner rule as profiles. + +CREATE TABLE IF NOT EXISTS nostr.addressable_latest ( + pubkey String, + kind UInt16, + d_tag String, + latest AggregateFunction(argMin, String, Tuple(Int64, String)), + last_indexed_at SimpleAggregateFunction(max, DateTime) +) ENGINE = AggregatingMergeTree +ORDER BY (pubkey, kind, d_tag); + +CREATE MATERIALIZED VIEW IF NOT EXISTS nostr.addressable_latest_mv TO nostr.addressable_latest +AS SELECT + pubkey, + kind, + arrayElement(arrayFilter(t -> t[1] = 'd', tags), 1)[2] AS d_tag, + argMinState(id, CAST((-toInt64(created_at), id), 'Tuple(Int64, String)')) AS latest, + max(indexed_at) AS last_indexed_at +FROM nostr.events_local +WHERE (kind >= 10000 AND kind < 20000) OR (kind >= 30000 AND kind < 40000) +GROUP BY pubkey, kind, d_tag; + +-- ============================================================================= +-- VERIFICATION +-- ============================================================================= + +SELECT 'Migration 017 objects' AS status, name, engine, metadata_modification_time +FROM system.tables +WHERE database = 'nostr' + AND name IN ( + 'profile_latest', 'profile_latest_mv', + 'zap_receipts_by_target', 'zap_receipts_by_target_mv', + 'addressable_latest', 'addressable_latest_mv' + ) +ORDER BY name; diff --git a/docs/migrations/rollback/017_drop_preview_compact_lookups.sql b/docs/migrations/rollback/017_drop_preview_compact_lookups.sql new file mode 100644 index 0000000..fa472f0 --- /dev/null +++ b/docs/migrations/rollback/017_drop_preview_compact_lookups.sql @@ -0,0 +1,32 @@ +-- Rollback: 017_preview_compact_lookups +-- Description: Remove the compact preview lookup tables and the materialized +-- views that feed them from events_local inserts. +-- Date: 2026-10-10 +-- +-- To run: +-- just ch-migrate docs/migrations/rollback/017_drop_preview_compact_lookups.sql +-- +-- IMPORTANT: +-- - First set PREVIEW_COMPACT_LOOKUPS=none and restart pensieve-preview, so +-- no preview read depends on these tables. +-- - Views are dropped before tables so no ingest flush writes to a missing +-- target. Dropping the views removes their per-insert cost immediately. +-- - This does not touch events_local or its projections. Re-applying +-- prepared/017 later requires a fresh plan and backfill. + +DROP VIEW IF EXISTS nostr.profile_latest_mv; +DROP VIEW IF EXISTS nostr.zap_receipts_by_target_mv; +DROP VIEW IF EXISTS nostr.addressable_latest_mv; + +DROP TABLE IF EXISTS nostr.profile_latest; +DROP TABLE IF EXISTS nostr.zap_receipts_by_target; +DROP TABLE IF EXISTS nostr.addressable_latest; + +SELECT 'Migration 017 objects remaining' AS status, count() AS objects +FROM system.tables +WHERE database = 'nostr' + AND name IN ( + 'profile_latest', 'profile_latest_mv', + 'zap_receipts_by_target', 'zap_receipts_by_target_mv', + 'addressable_latest', 'addressable_latest_mv' + ); diff --git a/docs/preview_compact_lookups.md b/docs/preview_compact_lookups.md new file mode 100644 index 0000000..caf9aa6 --- /dev/null +++ b/docs/preview_compact_lookups.md @@ -0,0 +1,276 @@ +# Compact ClickHouse lookups for pensieve-preview + +nstr.to (`pensieve-preview`) answers profile, zap and naddr lookups by +filtering `events_local` on `kind`. `events_local` is +`ReplacingMergeTree(indexed_at) ORDER BY id` with no partitions, so without +the full-row `events_by_kind` projection (about 1.33 TiB) those reads become +table scans: +- sampled profile lookups took 21–30 s; +- an event-zap lookup took more than 60 s. + +This change adds three small lookup tables, keyed the way preview reads them. +A reversible flag selects them, and the legacy reads stay available for +comparison and rollback. Removing the projection is **not** part of this +change. It stays a separate manual operation after validation (see +[`ops/RUNBOOK.md`](../ops/RUNBOOK.md#preview-compact-lookups)). + +| Piece | Location | +|-------|----------| +| Tables and materialized views | [`docs/migrations/prepared/017_preview_compact_lookups.sql`](migrations/prepared/017_preview_compact_lookups.sql) | +| Rollback | [`docs/migrations/rollback/017_drop_preview_compact_lookups.sql`](migrations/rollback/017_drop_preview_compact_lookups.sql) | +| Preview reads and flag | `crates/pensieve-preview/src/lookups/{mod,compact}.rs`, `PREVIEW_COMPACT_LOOKUPS` | +| Operator tool | `preview-lookups` (`crates/pensieve-preview/src/bin/preview-lookups.rs`) | +| Integration tests | `crates/pensieve-preview/tests/compact_lookups.rs` (`just ch-test-up && just test-clickhouse`) | + +## Which preview reads depended on the projection + +| Read | Legacy query | Compact replacement | +|------|--------------|---------------------| +| `fetch_profile`, `fetch_profiles`, `fetch_display_names` | `argMax(..., created_at)` over `kind = 0 AND pubkey IN (...)` | `profile_latest` | +| `fetch_engagement` zap totals | `events_local` (kind 9735, `arrayExists` e-tag) `JOIN zap_amounts_data` | `zap_receipts_by_target` | +| `fetch_replaceable` (naddr) | `kind = ? AND pubkey = ? AND d_tag = ? ORDER BY created_at DESC LIMIT 1` | `addressable_latest` + `events_local` by id | + +Reaction, comment and repost counts already use dedicated tables. +`fetch_event_by_id` is a primary-key read. Neither needs the projection. + +The legacy naddr query selects the MATERIALIZED `d_tag` column. A +`SELECT *` projection does not store MATERIALIZED columns. On ClickHouse +24.8 that query is therefore served by the base table and the +`idx_pubkey` bloom filter, not by `events_by_kind`. Production should confirm +this with `system.query_log.projections`. Either way, a pubkey bloom filter +over an id-ordered table reads every granule holding any row from that +author, which is slow for prolific authors. + +## Design + +### Correctness model + +Correctness is guaranteed before merges as well as after, and none of it +depends on background merges. Each read collapses duplicate rows or states +itself: `argMinMerge` for profiles and naddr, and `GROUP BY receipt_id` for +zaps. As a result: +- replayed inserts, which the ingester produces when it retries a segment, never inflate results; +- events arriving out of order never change the winner; +- overlap between backfill and live ingestion is harmless. + +`ReplacingMergeTree`/`AggregatingMergeTree` merges only make the tables +smaller. The integration tests check this by asserting every lookup twice: +once with merges stopped and duplicates physically present, and once after +`OPTIMIZE ... FINAL`. + +No view joins another table. An insert-triggered view only reacts to inserts +into its source table, so it would miss later changes on the other side of a +join. + +### `profile_latest` + +`AggregatingMergeTree ORDER BY pubkey`, one aggregate state per pubkey: + +```sql +latest = argMinState((id, content, created_at), (-toInt64(created_at), id)) +``` + +- **Winner:** the highest `created_at` wins. On a tie, the lowest event id wins (NIP-01 replaceable semantics). +- **Idempotent and order-independent:** `argMin` is both, so duplicates, out-of-order arrivals and repeated backfill shards cannot change the result. +- **Consistent payload:** `id`, `content` and `created_at` always come from the same event. +- **Pinned types:** explicit `CAST`s pin the tuple types so the view's output matches the column type exactly. A mismatch would make the view throw on every insert. A test asserts the column types. + +### `zap_receipts_by_target` + +`ReplacingMergeTree(parser_version) ORDER BY (target_event_id, receipt_id)`. +It stores one row per distinct non-empty `e` tag of each kind 9735 receipt. + +**How the amount is computed:** +- The amount comes from the receipt's own `bolt11` tag, using **exactly the migration 009 rules**: + - the amount is taken from the first `bolt11` tag; + - multipliers `m`/`u`/`n`/`p` only; + - values above 1 000 000 000 msats become 0. +- `amount_parsed` records 009's inclusion rule: some `bolt11` tag matches `^lnbc[0-9]+[munp]1`. + +The read collapses rows per receipt and takes the newest parser version: + +```sql +SELECT countIf(p = 1), sumIf(a, p = 1) FROM ( + SELECT receipt_id, argMax(amount_msats, parser_version) a, argMax(amount_parsed, parser_version) p + FROM zap_receipts_by_target WHERE target_event_id = ? GROUP BY receipt_id) +``` + +**Amounts arriving before or after receipts.** Today there is no separate +amount stream: `zap_amounts_data` is filled by an insert view over the same +`events_local` insert block as the receipt. The amount is part of the signed +receipt, so deriving it from the receipt row removes any ordering between +amount and receipt. Later rows in `zap_amounts_data` have no effect; a test +covers this. + +**Corrections.** A future parser fix ships as a new view writing +`parser_version = 2` alongside v1, followed by a re-backfill. `argMax` picks +the newest version whichever arrives first. Tests cover v2 arriving before +and after v1, both before and after merges. Once v2 is fully backfilled, drop +the v1 view. + +### `addressable_latest` + +`AggregatingMergeTree ORDER BY (pubkey, kind, d_tag)` covering kinds +10000–19999 and 30000–39999. +- **Winner rule:** the same `argMin` rule as profiles, but storing only the winning event id. The preview then reads the row from `events_local` in one query (`WHERE id IN (SELECT argMinMerge(latest) ...)`), which is a primary-key read. +- **`d_tag`:** uses the same expression as the `events_local.d_tag` MATERIALIZED column, so lookups match the legacy `d_tag = ?` filter exactly. +- **Uncovered kinds:** naddr kinds outside the covered ranges use the legacy query. + +### Ingest safety + +The new views run inside every `events_local` flush. Production uses +`async_insert=1, wait_for_async_insert=0`, so the ingester never sees a flush +error. A view that throws would fail the whole flush, and the rows would be +lost silently. + +Precautions: +- Every view expression is non-throwing: `OrZero` conversions, array indexing that returns defaults, and no fallible casts. +- An integration test pushes malformed tags through production async-insert settings and asserts that every row lands and `system.asynchronous_insert_log` shows no failures. +- Monitoring queries are in the runbook. + +## Behaviour differences from the legacy reads + +| Area | Legacy | Compact | +|------|--------|---------| +| Profile `created_at` ties | Arbitrary event; `id` and `content` come from separate `argMax` calls and can come from **different** events | Lowest event id wins; `id` and `content` always from one event | +| naddr `created_at` ties | Arbitrary (`ORDER BY created_at DESC LIMIT 1`) | Lowest event id wins | +| Zap counts and amounts | Unmerged duplicate rows in `events_local` or `zap_amounts_data` multiply `count()` and `sum()` | Each receipt is counted once per target | +| Zap inclusion and amount rules | Migration 009 | Identical (tested for parity after merges) | +| Deletions (kind 5) | Ignored | Ignored (unchanged) | +| Freshness | Insert-time | Insert-time (the views run in the same flush) | + +## Operator tool: `preview-lookups` + +```bash +preview-lookups [--clickhouse-url ... --clickhouse-database nostr] \ + [--max-threads 2 --max-memory-gib 8 --max-execution-secs 3600] \ + [--checkpoint-dir data/preview-lookups] +``` + +Connection settings also come from `CLICKHOUSE_URL`, `CLICKHOUSE_DATABASE`, +`CLICKHOUSE_USER` and `CLICKHOUSE_PASSWORD`. Every query carries the +`max_threads`, `max_memory_usage`, external aggregation/sort spill limits and +`max_execution_time` settings. + +| Command | Purpose | +|---------|---------| +| `plan` | Reads source statistics and an hourly histogram through `events_by_kind`. It writes `//plan.json` with contiguous `created_at` shards; the first and last are open-ended, so `created_at = 0` and future timestamps are covered. It also prints the disk estimate. | +| `backfill` | Runs remaining shards (resumable, idempotent). See the guards below. | +| `status` | Shards done, parts, rows, bytes and freshness lag per table. | +| `validate` | Compares compact and legacy reads with a Rust oracle computed from source events, at a fixed cutoff. Exits non-zero on any unexplained difference. | +| `bench` | Lookup latency with `optimize_use_projections=0` and `use_query_cache=0`, plus storage. | + +### Backfill guards + +- **Gap preflight.** The plan records the view's creation time `T0`. A + backfill refuses to start unless all of these hold: + - the view still has that creation time; + - `now >= T0 + grace` (default 10 minutes); + - no `INSERT` into `events_local` older than `T0` is still running; + - no async-insert buffer for `events_local` was first written before `T0`. + + So every row whose insert could have missed the view is already committed + before the first shard reads. Every later insert is captured by the view, + including inserts carrying old `created_at` values for shards that are + already done. Overlap is harmless, as described above. `--gap-pass` adds an + optional belt-and-braces re-read of rows with `indexed_at` within + `--gap-margin-mins` of `T0`; it reads the whole target-kind slice. +- **Projection enforcement.** Shards run with `optimize_use_projections=1` + and `force_optimize_projection=1`. After each shard the tool also checks + `system.query_log`: + - `events_by_kind` must appear in `projections`; + - `read_rows` must stay under `--max-read-amplification` × the planned rows + (plus 1M rows of slack). + + This catches parts that lack the projection and would silently be scanned + in full. `--allow-full-scan` with `--shard-by id-prefix` is only for rebuilds + after the projection is gone; each shard then reads a primary-key range of + the whole table. +- **Disk guard.** Before each shard, the unreserved space on the table's disk + must cover the remaining share of the peak estimate plus `--min-free-gib` + (default 100). Otherwise the run stops cleanly and can be resumed. +- **Merge backlog.** While the target has more than `--max-target-parts` + active parts (default 300), the tool waits for merges. After 30 minutes it + stops cleanly. +- **Runtime.** `--max-runtime-mins`, `--max-shards` and `--delay-secs` bound + each session. Each finished shard writes a JSON checkpoint containing its + `system.query_log` entry. +- **Progress.** Each shard logs done/total, percent, rows read and written, + duration and ETA. + +### Disk estimate + +`plan` prints the following for each target, plus a combined peak compared +with the disk's unreserved space: + +- **bytes per row:** `fixed_bytes × 0.6 + payload_bytes × r`. + - `fixed_bytes` is the hex keys and fixed-width columns. 0.6 is the assumed + compression ratio for random hex. + - `payload_bytes` is the average `content` (profiles) or `d` tag (naddr) + length, measured from the source. + - `r` is `events_local.content`'s actual compression ratio from + `system.columns`. +- **final size:** `output_rows × bytes_per_row`, where output rows are the + merged keys (`uniq`). +- **pre-merge size:** the states written before merges collapse them, per + shard `min(shard_rows, distinct_keys)`. For zaps this equals the target rows. +- **peak:** `2 × pre-merge`. A merge writes its output while its inputs, and + then the outdated parts until `old_parts_lifetime` passes, still exist. + +This is an estimate. Calibrate it during the first production shards by +comparing `status` (bytes on disk per row) with the plan, and keep +`--min-free-gib` as a hard floor. + +## Validation and benchmark + +- **`validate`** + - Samples keys from **`events_local`**, so keys missing from the compact + tables are detected. It uses a hashed random subset plus the hottest keys. + - Compares each key at a consistent cutoff: any key with a source row + indexed after `now - --cutoff-margin-secs` is skipped. + - The compact read must equal the oracle exactly. + - Legacy differences are classified as `legacy_tie_break`, + `legacy_id_content_mix`, `legacy_duplicate_inflation`, + `legacy_amount_row_missing` or `unexplained`. +- **`bench`** + - Measures single profile, 50-pubkey batch, display names, zaps, the + hottest zap targets and naddr. Measured queries run with + `optimize_use_projections=0` and `use_query_cache=0`. + - Every run uses fresh keys (a new seed per run, with disjoint key pools). + - `--drop-caches` drops the mark and uncompressed caches before every query. + This affects the whole server; use it only in a quiet window. + - Server duration and rows/bytes read come from `system.query_log` + (`log_comment` tags). + - `--include-legacy` measures the legacy reads under the same settings with + a capped timeout. + - The report states `under target` per lookup against `--target-ms` + (default 1000) from measured numbers only. The OS page cache cannot be + dropped from SQL, so state whether production runs were warm. + +Preview queries carry `log_comment = 'pensieve-preview::'`, +so `system.query_log` gives production latency per lookup and source during +rollout. + +## Requires production validation + +The checks below can only be done on the production server. Locally, the +integration suite and an end-to-end CLI run against ClickHouse 24.8.14.39 (the +production build) prove correctness and mechanics only. + +1. **Schema:** `SHOW CREATE TABLE nostr.events_local` and + `system.projection_parts` show which projections exist. Is + `events_by_author` present? Confirm that every active part has + `events_by_kind` before relying on it for the backfill. +2. **Sizing:** row and key counts per target from `plan`, especially + `addressable_latest` (kind 30078 app data may be large), the peak estimate + against `/data` unreserved space, and calibration after the first shards. +3. **Backfill impact:** runtime and impact on live ingest and queries at the + chosen `--max-threads`/`--max-memory-gib`. +4. **Ingest overhead:** flush latency and errors after the views are created + (`system.asynchronous_insert_log`, `system.query_views_log`). +5. **Latency:** `bench` with projections off, query cache off and + `--drop-caches` in a quiet window. The sub-second target is met only if + this run shows it. +6. **Other projection consumers:** the `system.query_log` audit in the + runbook, covering `pensieve-serve` kinds/stats endpoints, negentropy + seeding and Grafana. diff --git a/env.local.example b/env.local.example index 65a44f1..bec1801 100644 --- a/env.local.example +++ b/env.local.example @@ -28,3 +28,5 @@ RELAY_DB_PATH=./data/relay-stats.db PREVIEW_BIND_ADDR=127.0.0.1:8081 PREVIEW_BASE_URL=http://localhost:8081 PREVIEW_SITE_NAME=Nostr Preview +# Compact ClickHouse lookups (migration 017): none, all, or profiles,zaps,addressable +# PREVIEW_COMPACT_LOOKUPS=none diff --git a/env.production.example b/env.production.example index b665c09..7ac672c 100644 --- a/env.production.example +++ b/env.production.example @@ -48,6 +48,11 @@ PREVIEW_SITE_NAME=Nostr Preview # Path to VIP pubkeys file — one pubkey per line, hex or npub (optional). # See ops/production/.vip.example for format. # PREVIEW_VIP_FILE=/home/pensieve/pensieve/.vip +# Which preview lookups read the compact ClickHouse tables (migration 017) +# instead of events_local: none (default), all, or a comma list of +# profiles,zaps,addressable. Reversible with a restart. See +# docs/preview_compact_lookups.md before enabling. +# PREVIEW_COMPACT_LOOKUPS=none # ═══════════════════════════════════════════════════════════════════════════ # Ingester diff --git a/justfile b/justfile index cf3564e..f7a43fc 100644 --- a/justfile +++ b/justfile @@ -160,6 +160,10 @@ coverage-preview: coverage-preview-summary: cargo llvm-cov --lib -p pensieve-preview +# Run the compact preview lookup operator tool (plan/backfill/status/validate/bench) +run-preview-lookups *ARGS: + cargo run --bin preview-lookups -- {{ARGS}} + # Run the relay cleanup tool run-relay-cleanup *ARGS: cargo run --bin relay-cleanup -- {{ARGS}} @@ -270,6 +274,27 @@ ch-tables: @docker exec -i {{CH_CONTAINER}} clickhouse-client --database {{CH_DB}} \ --query "SELECT name, engine FROM system.tables WHERE database = '{{CH_DB}}' ORDER BY engine, name" +# Disposable ClickHouse for integration tests (same 24.8 line as production) +CH_TEST_CONTAINER := env_var_or_default("CH_TEST_CONTAINER", "pensieve-ch-test") +CH_TEST_PORT := env_var_or_default("CH_TEST_PORT", "18123") + +# Start a disposable ClickHouse 24.8 for integration tests +ch-test-up: + @docker run -d --rm --name {{CH_TEST_CONTAINER}} -p {{CH_TEST_PORT}}:8123 \ + -e CLICKHOUSE_SKIP_USER_SETUP=1 --ulimit nofile=262144:262144 \ + clickhouse/clickhouse-server:24.8 >/dev/null + @until curl -sf http://localhost:{{CH_TEST_PORT}}/ping >/dev/null; do sleep 1; done + @echo "✓ ClickHouse test server on http://localhost:{{CH_TEST_PORT}}" + +# Stop the disposable ClickHouse test server +ch-test-down: + @docker rm -f {{CH_TEST_CONTAINER}} >/dev/null && echo "✓ ClickHouse test server removed" + +# Run ClickHouse integration tests (start a server with `just ch-test-up`) +test-clickhouse: + PENSIEVE_TEST_CLICKHOUSE_URL=${PENSIEVE_TEST_CLICKHOUSE_URL:-http://localhost:{{CH_TEST_PORT}}} \ + cargo test -p pensieve-preview --test compact_lookups -- --ignored + # ============================================================================ # Utilities # ============================================================================ diff --git a/ops/RUNBOOK.md b/ops/RUNBOOK.md index e40ab03..6183cd6 100644 --- a/ops/RUNBOOK.md +++ b/ops/RUNBOOK.md @@ -380,6 +380,176 @@ pulling this change, or services break. On the box, after `git pull`: systemctl status pensieve pensieve-api pensieve-preview pensieve-ingest ``` +## Preview compact lookups + +Moves nstr.to's profile, zap and naddr reads off the `events_by_kind` +projection and onto small lookup tables. Design, correctness model and +behaviour differences: [`docs/preview_compact_lookups.md`](../docs/preview_compact_lookups.md). +Every step below can be undone. **Dropping the projection is not one of these +steps** (see the end of this section). + +Run the tool from the checkout as the `pensieve` user, with +`/etc/pensieve/pensieve.env` loaded (`CLICKHOUSE_URL`, `CLICKHOUSE_DATABASE`): + +```bash +cargo build --release --bin preview-lookups +PL="target/release/preview-lookups --checkpoint-dir /home/pensieve/preview-lookups" +``` + +### 1. Preflight (read-only) + +```bash +just ch-query "SELECT version()" +just ch-query "SHOW CREATE TABLE nostr.events_local" +just ch-query "SELECT name, count() parts, sum(rows) rows, formatReadableSize(sum(bytes_on_disk)) bytes FROM system.projection_parts WHERE database = 'nostr' AND table = 'events_local' AND active GROUP BY name ORDER BY name" +# Every active part must carry events_by_kind (expect 0): +just ch-query "SELECT count() FROM system.parts p WHERE database = 'nostr' AND table = 'events_local' AND active AND name NOT IN (SELECT parent_name FROM system.projection_parts WHERE database = 'nostr' AND table = 'events_local' AND name = 'events_by_kind' AND active)" +just ch-query "SELECT name, formatReadableSize(unreserved_space) unreserved, formatReadableSize(total_space) total FROM system.disks" +just ch-query "SELECT count() FROM system.mutations WHERE database = 'nostr' AND table = 'events_local' AND is_done = 0" +just ch-query "SELECT count() FROM system.asynchronous_insert_log WHERE event_date = today()" +``` + +Do not continue if `events_by_kind` is missing from any part or a mutation +is pending. Record the outputs in the change log. + +### 2. Create the tables and views + +```bash +just ch-migrate docs/migrations/prepared/017_preview_compact_lookups.sql +``` + +This creates empty tables and three insert-triggered views on `events_local`. +For the next 30 minutes, watch the ingest flushes and the views: + +```bash +just ch-query "SELECT status, count(), any(exception) FROM system.asynchronous_insert_log WHERE event_time > now() - INTERVAL 30 MINUTE AND database = 'nostr' AND table = 'events_local' GROUP BY status" +just ch-query "SELECT view_name, status, count(), any(exception) FROM system.query_views_log WHERE event_time > now() - INTERVAL 30 MINUTE AND view_name LIKE 'nostr.%' GROUP BY view_name, status" +journalctl -u pensieve-ingest --since "30 min ago" | grep -i clickhouse +``` + +**Abort:** if any flush status is not `Ok`, or any view reports an +exception, run the rollback SQL immediately (step 7). A failing view drops +ingest rows silently under `wait_for_async_insert=0`. + +### 3. Plan and estimate disk + +```bash +$PL plan --max-rows-per-shard 2000000 --min-free-gib 100 +``` + +For each target, check source rows, keys, final and peak size, and the +combined peak against `/data` unreserved space. If `addressable` is +unexpectedly large, plan and backfill `--target profiles --target zaps` first. + +### 4. Backfill (while `events_by_kind` exists) + +Wait 10 minutes after step 2. The tool enforces this and the other gap checks. +Then run bounded sessions; each session resumes where the last stopped: + +```bash +$PL --max-threads 2 --max-memory-gib 8 backfill --max-runtime-mins 120 --delay-secs 10 --min-free-gib 100 +$PL status +``` + +- **Exit code 2** means the run stopped early (runtime, disk guard, or merge + backlog). Re-run the same command later. +- **Error on projection or read amplification:** the tool stops if a shard + did not use `events_by_kind` or read far more rows than planned. + Investigate before continuing, and never re-run with `--allow-full-scan` on + production while the projection exists. +- **Calibrate:** after the first few shards, compare `status` bytes per row + with the plan's estimate. +- **Optional:** add `--gap-pass` on the final session to re-read rows indexed + around the views' creation time. + +### 5. Validate and benchmark + +```bash +$PL validate --sample 1000 --hot 100 --out /home/pensieve/preview-lookups/validate.json +$PL bench --samples 50 --out /home/pensieve/preview-lookups/bench-warm.json +# Quiet window only (drops server-wide caches before every query): +$PL bench --samples 50 --drop-caches --include-legacy --legacy-samples 3 --out /home/pensieve/preview-lookups/bench-cold.json +``` + +- **`validate`** must report `unexplained differences: 0`. Legacy + `tie_break`, `id_content_mix`, `duplicate_inflation` and + `amount_row_missing` are known legacy defects (see the design doc). +- **`bench`** results are what decide the sub-second target. Record p95 and + max per lookup, and storage (compact tables vs `events_by_kind`), in the + change log. + +### 6. Switch preview reads (staged) + +Set `PREVIEW_COMPACT_LOOKUPS` in `/etc/pensieve/pensieve.env` one lookup at a +time, then `sudo systemctl restart pensieve-preview`: + +1. `profiles` +2. `profiles,zaps` +3. `all` + +Between steps, compare live latency per lookup and source: + +```bash +just ch-query "SELECT log_comment, count() n, quantile(0.5)(query_duration_ms) p50, quantile(0.95)(query_duration_ms) p95, max(query_duration_ms) max FROM system.query_log WHERE event_time > now() - INTERVAL 1 HOUR AND type = 'QueryFinish' AND startsWith(log_comment, 'pensieve-preview:') GROUP BY log_comment ORDER BY log_comment" +journalctl -u pensieve-preview --since "1 hour ago" | grep -c "slow preview lookup" +``` + +Spot-check a few profiles, zapped notes and articles on nstr.to (`.json` +routes show raw values). The 5-minute HTML cache and CDN delay what you see. + +### 7. Rollback + +- **Reads:** set `PREVIEW_COMPACT_LOOKUPS=none` and restart + `pensieve-preview`. The legacy reads are untouched and use the projection + as before. +- **Tables and views** (only after reads are rolled back): + + ```bash + just ch-migrate docs/migrations/rollback/017_drop_preview_compact_lookups.sql + ``` + + Re-applying 017 later needs a fresh `plan --discard-progress` and backfill. + +### 8. Projection consumers (before any removal) + +The preview is not the only reader that may depend on `events_by_kind`. Known +candidates from the repo: +- `pensieve-serve` `routes/kinds.rs` and `routes/stats.rs` (kind-filtered + scans); +- negentropy `seed_from_clickhouse` (a `created_at` window); +- Grafana dashboards, which live only on the instance. + +With the preview fully switched, audit actual use over at least the 7-day +`query_log` retention. `preview-lookups` tags its own queries and is +excluded: + +```bash +just ch-query "SELECT user, splitByChar(':', log_comment)[1] AS client, normalized_query_hash, any(substring(query, 1, 160)) AS example, count() runs, round(avg(query_duration_ms)) avg_ms, max(event_time) last_seen FROM system.query_log WHERE event_date >= today() - 7 AND type = 'QueryFinish' AND has(projections, 'nostr.events_local.events_by_kind') AND NOT startsWith(log_comment, 'preview-lookups') GROUP BY user, client, normalized_query_hash ORDER BY runs DESC LIMIT 100" +``` + +For each consumer listed, measure it with +`SETTINGS optimize_use_projections = 0` (and `use_query_cache = 0`) before +deciding. The dashboard queries were previously measured at about 9.2 s for +14 queries without the projection. + +### Projection removal (separate, manual) + +**Not part of this rollout.** Only consider it after all of these hold: +- every backfill is complete; +- `validate` is clean; +- `bench` meets the target; +- the preview has run on `all` for an agreed period; +- the consumer audit above is clear or each consumer is accepted; +- the removal is written up as its own prepared migration, following the + `prepared/016` pattern (preflight, asynchronous `DROP PROJECTION`, rollback + with free-space check). + +After removal, the legacy preview reads become slow again, so rolling back +reads means restoring the projection first. Future rebuilds of the compact +tables (for example a zap parser correction) then need +`plan --allow-full-scan --shard-by id-prefix`. That runs full-table reads in +256 bounded shards; plan the I/O accordingly. + ## Grafana Provisioning is no longer in the repo. Configure the Prometheus datasource From cad1b67189c2170d1397c42c032fa34de55ca7fa Mon Sep 17 00:00:00 2001 From: Jeff Gardner <202880+erskingardner@users.noreply.github.com> Date: Sat, 10 Oct 2026 14:25:47 -0500 Subject: [PATCH 2/6] Harden preview-lookups validation and backfill guards from review - validate: re-check source freshness after the compact and legacy reads so inserts that land mid-validation are skipped instead of reported as differences; a target whose sampled keys were all skipped is inconclusive and fails the run - backfill: when the projection is required, a shard without a system.query_log entry is not checkpointed (fail closed) - gap pass: size it first and apply the runtime, disk and merge-backlog guards like any shard; disk guard arithmetic is overflow-safe - regression tests for each case Co-Authored-By: Claude Opus 5.5 --- .../src/bin/preview-lookups.rs | 13 +- .../pensieve-preview/src/lookups/backfill.rs | 120 ++++++++- .../pensieve-preview/src/lookups/validate.rs | 247 ++++++++++++++++-- .../pensieve-preview/tests/compact_lookups.rs | 159 ++++++++++- docs/preview_compact_lookups.md | 13 +- ops/RUNBOOK.md | 4 +- 6 files changed, 512 insertions(+), 44 deletions(-) diff --git a/crates/pensieve-preview/src/bin/preview-lookups.rs b/crates/pensieve-preview/src/bin/preview-lookups.rs index 421220c..4597a4e 100644 --- a/crates/pensieve-preview/src/bin/preview-lookups.rs +++ b/crates/pensieve-preview/src/bin/preview-lookups.rs @@ -308,7 +308,11 @@ async fn run(args: Args) -> Result { break; } if gap_pass { - backfill::run_gap_pass( + let options = BackfillOptions { + max_runtime: max_runtime.map(|m| m.saturating_sub(started.elapsed())), + ..options + }; + let stopped = backfill::run_gap_pass( &client, &plan, &checkpoints, @@ -316,6 +320,11 @@ async fn run(args: Args) -> Result { &options, ) .await?; + println!("{target} gap pass: {}", serde_json::to_string(&stopped)?); + if stopped != StopReason::Complete { + complete = false; + break; + } } } if complete { @@ -357,7 +366,7 @@ async fn run(args: Args) -> Result { let report = validate::run(&client, &database, &options).await?; print!("{}", report.render()); write_report(out.as_ref(), &report)?; - Ok(if report.unexplained() == 0 { + Ok(if report.passed() { ExitCode::SUCCESS } else { ExitCode::FAILURE diff --git a/crates/pensieve-preview/src/lookups/backfill.rs b/crates/pensieve-preview/src/lookups/backfill.rs index 90addab..83f00a6 100644 --- a/crates/pensieve-preview/src/lookups/backfill.rs +++ b/crates/pensieve-preview/src/lookups/backfill.rs @@ -298,13 +298,35 @@ pub async fn run_gap_pass( checkpoints: &CheckpointDir, margin: Duration, options: &BackfillOptions, -) -> Result { +) -> Result { + if options.max_runtime.is_some_and(|m| m.is_zero()) { + return Ok(StopReason::MaxRuntime); + } let margin = margin.as_secs() as u32; let range = ShardRange::IndexedAt { start: plan.view_created_at.saturating_sub(margin), end: plan.view_created_at.saturating_add(margin), }; let insert_client = projection_client(client, plan.require_projection); + + // The pass is guarded like any shard: size it, then check disk and merges. + let estimated_rows = insert_client + .query(&format!( + "SELECT count() FROM {}.events_local WHERE {} AND ({})", + plan.database, + plan.target.kind_predicate(), + range.predicate() + )) + .fetch_one::() + .await + .with_context(|| format!("sizing {} gap-closure pass", plan.target))?; + if let Some(refusal) = disk_guard(client, plan, estimated_rows, options).await? { + return Ok(StopReason::DiskGuard(refusal)); + } + if let Some(backlog) = wait_for_merges(client, plan, options).await? { + return Ok(StopReason::PartsBacklog(backlog)); + } + // Read amplification is expected here (indexed_at is not a key column). let options = BackfillOptions { max_read_amplification: f64::INFINITY, @@ -316,17 +338,18 @@ pub async fn run_gap_pass( plan, usize::MAX, &range, - 0, + estimated_rows, &options, ) .await?; checkpoints.save_gap_pass(&checkpoint)?; tracing::info!( target = %plan.target, + estimated_rows, written_rows = checkpoint.query_log.as_ref().map_or(0, |q| q.written_rows), "gap-closure pass complete" ); - Ok(checkpoint) + Ok(StopReason::Complete) } fn report( @@ -371,14 +394,8 @@ async fn run_shard( let elapsed_ms = started.elapsed().as_millis() as u64; let query_log = ops::query_log_entry(log_client, &query_id, QUERY_LOG_TIMEOUT).await?; - if let Some(entry) = &query_log { - check_projection_use(plan, index, estimated_rows, entry, options)?; - } else { - tracing::warn!( - query_id, - "no system.query_log entry; projection use not verified" - ); - } + check_projection_use(plan, index, estimated_rows, query_log.as_ref(), options) + .with_context(|| format!("query id {query_id}"))?; Ok(ShardCheckpoint { index, @@ -390,16 +407,30 @@ async fn run_shard( }) } +/// Verify a shard was served by `events_by_kind`. Fails closed: when the +/// projection is required and the shard has no `system.query_log` entry, the +/// shard is not checkpointed (its rows are harmless and a re-run redoes it). fn check_projection_use( plan: &Plan, index: usize, estimated_rows: u64, - entry: &QueryLogEntry, + entry: Option<&QueryLogEntry>, options: &BackfillOptions, ) -> Result<()> { if !plan.require_projection { + if entry.is_none() { + tracing::warn!(target = %plan.target, shard = index, "no system.query_log entry for shard"); + } return Ok(()); } + let Some(entry) = entry else { + bail!( + "{} shard {index} has no system.query_log entry, so projection use cannot be \ + verified; not checkpointing it. Rows it wrote are harmless. Check that \ + query_log is enabled and flushing, then re-run to retry the shard.", + plan.target + ); + }; let projection = format!("{}.events_local.events_by_kind", plan.database); if !entry.projections.iter().any(|p| p == &projection) { bail!( @@ -434,13 +465,14 @@ async fn disk_guard( } else { (remaining_rows as f64 / plan.source.source_rows as f64).min(1.0) }; - let needed = (e.peak_bytes as f64 * remaining_fraction) as u64 + options.min_free_bytes; + let needed = + ((e.peak_bytes as f64 * remaining_fraction) as u64).saturating_add(options.min_free_bytes); if disk.unreserved_space < needed { return Ok(Some(format!( "disk {} has {} unreserved; remaining backfill needs ~{} plus {} reserve", disk.name, ops::human_bytes(disk.unreserved_space), - ops::human_bytes(needed - options.min_free_bytes), + ops::human_bytes(needed.saturating_sub(options.min_free_bytes)), ops::human_bytes(options.min_free_bytes) ))); } @@ -475,3 +507,63 @@ async fn wait_for_merges( tokio::time::sleep(PARTS_POLL).await; } } + +#[cfg(test)] +mod tests { + use super::*; + use crate::lookups::plan::{DiskEstimate, PLAN_FORMAT_VERSION, ShardBy, SourceStats}; + use crate::lookups::target::Target; + + fn plan(require_projection: bool) -> Plan { + Plan { + format_version: PLAN_FORMAT_VERSION, + database: "db".into(), + target: Target::Profiles, + shard_by: ShardBy::CreatedAt, + require_projection, + view_created_at: 0, + planned_at: 0, + source: SourceStats::default(), + estimate: DiskEstimate::default(), + shards: Vec::new(), + } + } + + fn entry(projections: &[&str], read_rows: u64) -> QueryLogEntry { + QueryLogEntry { + kind: "QueryFinish".into(), + read_rows, + projections: projections.iter().map(|p| p.to_string()).collect(), + ..QueryLogEntry::default() + } + } + + #[test] + fn missing_query_log_fails_closed_when_projection_required() { + let options = BackfillOptions::default(); + assert!(check_projection_use(&plan(true), 3, 100, None, &options).is_err()); + assert!(check_projection_use(&plan(false), 3, 100, None, &options).is_ok()); + } + + #[test] + fn projection_and_read_amplification_checks() { + let options = BackfillOptions::default(); + let used = "db.events_local.events_by_kind"; + assert!( + check_projection_use(&plan(true), 0, 100, Some(&entry(&[used], 50)), &options).is_ok() + ); + assert!( + check_projection_use(&plan(true), 0, 100, Some(&entry(&[], 50)), &options).is_err() + ); + assert!( + check_projection_use( + &plan(true), + 0, + 100, + Some(&entry(&[used], 2_000_000)), + &options + ) + .is_err() + ); + } +} diff --git a/crates/pensieve-preview/src/lookups/validate.rs b/crates/pensieve-preview/src/lookups/validate.rs index e26fdce..673f5d8 100644 --- a/crates/pensieve-preview/src/lookups/validate.rs +++ b/crates/pensieve-preview/src/lookups/validate.rs @@ -6,15 +6,19 @@ //! //! 1. reads every source version and computes the expected answer with the //! independent Rust [`oracle`](super::oracle); -//! 2. skips the key if any source row was indexed after the cutoff, so both -//! sides are compared at a consistent point in time; +//! 2. skips the key if any source row was indexed after the cutoff — checked +//! both before and again after the compact and legacy reads, so a +//! concurrent insert cannot be misread as a difference; //! 3. requires the compact read (the exact function the preview uses) to //! equal the oracle — any difference is **unexplained**; //! 4. compares the legacy read and classifies differences as known legacy //! defects (timestamp tie-breaks, id/content from different rows, //! duplicate inflation, missing amount rows) or unexplained. +//! +//! A target whose sampled keys were all skipped is **inconclusive**, not a +//! pass. -use std::collections::{BTreeMap, BTreeSet, HashMap}; +use std::collections::{BTreeMap, BTreeSet, HashMap, HashSet}; use std::time::Duration; use anyhow::{Context, Result}; @@ -109,7 +113,8 @@ pub struct TargetReport { pub target: Target, /// Keys sampled. pub sampled: usize, - /// Keys skipped because source rows were indexed after the cutoff. + /// Keys skipped because source rows were indexed after the cutoff + /// (before or during validation). pub skipped_after_cutoff: usize, /// Compact-vs-source outcomes. pub compact: BTreeMap, @@ -155,6 +160,17 @@ impl TargetReport { } } + /// Keys whose compact read was compared with the source. + pub fn compared(&self) -> u64 { + self.compact.values().sum() + } + + /// Keys were sampled but none could be compared, so the target provides + /// no correctness evidence. + pub fn inconclusive(&self) -> bool { + self.sampled > 0 && self.compared() == 0 + } + /// Unexplained differences on either side. pub fn unexplained(&self) -> u64 { self.compact @@ -182,13 +198,27 @@ impl ValidationReport { self.targets.iter().map(TargetReport::unexplained).sum() } + /// Targets with sampled keys but no comparisons. + pub fn inconclusive_targets(&self) -> Vec { + self.targets + .iter() + .filter(|t| t.inconclusive()) + .map(|t| t.target) + .collect() + } + + /// No unexplained differences and every target compared at least one key. + pub fn passed(&self) -> bool { + self.unexplained() == 0 && self.inconclusive_targets().is_empty() + } + /// Text summary. pub fn render(&self) -> String { let mut out = format!("cutoff {} (seed {})\n", self.cutoff, self.seed); for t in &self.targets { out.push_str(&format!( - "{}: sampled {}, skipped after cutoff {}\n compact vs source: {:?}\n legacy vs source: {:?}\n", - t.target, t.sampled, t.skipped_after_cutoff, t.compact, t.legacy + "{}: sampled {}, compared {}, skipped after cutoff {}\n compact vs source: {:?}\n legacy vs source: {:?}\n", + t.target, t.sampled, t.compared(), t.skipped_after_cutoff, t.compact, t.legacy )); for d in &t.examples { out.push_str(&format!( @@ -201,6 +231,13 @@ impl ValidationReport { "unexplained differences: {}\n", self.unexplained() )); + let inconclusive = self.inconclusive_targets(); + if !inconclusive.is_empty() { + out.push_str(&format!( + "INCONCLUSIVE (no keys compared): {inconclusive:?}; lower --cutoff-margin-secs \ + only if ingest is quiet, or retry later\n" + )); + } out } } @@ -335,10 +372,11 @@ async fn validate_profiles(ctx: &Ctx<'_>) -> Result { let compact_rows = compact::fetch_profiles(ctx.compact, batch).await?; let legacy_rows = query::fetch_profiles(ctx.legacy, batch).await?; + let changed = profiles_changed_after(ctx.source, db, batch, ctx.cutoff).await?; for key in batch { let versions = by_key.get(key.as_str()).cloned().unwrap_or_default(); - if versions.iter().any(|r| r.indexed_at > ctx.cutoff) { + if changed.contains(key) || versions.iter().any(|r| r.indexed_at > ctx.cutoff) { report.skipped_after_cutoff += 1; continue; } @@ -457,18 +495,36 @@ async fn validate_zaps(ctx: &Ctx<'_>) -> Result { } } + // Read compact (and some legacy) results first, then re-check source + // freshness so keys changed during the reads are skipped. + let mut reads = Vec::new(); for key in batch { let receipts = by_target.get(key).cloned().unwrap_or_default(); if receipts.iter().any(|r| r.indexed_at > ctx.cutoff) { report.skipped_after_cutoff += 1; continue; } + let compact_totals = compact::fetch_zap_totals(ctx.compact, key).await?; + let legacy = if legacy_checked < o.legacy_sample { + legacy_checked += 1; + Some(query::fetch_zap_totals(ctx.legacy, key).await?) + } else { + None + }; + reads.push((key, receipts, compact_totals, legacy)); + } + let changed = zap_targets_changed_after(ctx.source, db, batch, ctx.cutoff).await?; + + for (key, receipts, compact_totals, legacy) in reads { + if changed.contains(key) { + report.skipped_after_cutoff += 1; + continue; + } let events: Vec = receipts.iter().map(|r| r.event()).collect(); let expected = oracle::zap_totals_by_target(&events) .remove(key.as_str()) .unwrap_or_default(); - let compact_totals = compact::fetch_zap_totals(ctx.compact, key).await?; let outcome = if compact_totals == expected { Outcome::Match } else { @@ -482,11 +538,9 @@ async fn validate_zaps(ctx: &Ctx<'_>) -> Result { o.max_examples, ); - if legacy_checked >= o.legacy_sample { + let Some(legacy) = legacy else { continue; - } - legacy_checked += 1; - let legacy = query::fetch_zap_totals(ctx.legacy, key).await?; + }; let outcome = if legacy == expected { Outcome::Match } else if legacy.zap_count > expected.zap_count && zap_duplicates(ctx, key).await? > 0 { @@ -590,8 +644,11 @@ async fn validate_addressable(ctx: &Ctx<'_>) -> Result { .push(row); } - for (pubkey, kind, d) in batch { - let label = format!("{kind}:{pubkey}:{d}"); + // Read compact (and some legacy) results first, then re-check source + // freshness so coordinates changed during the reads are skipped. + let mut reads = Vec::new(); + for coordinate in batch { + let (pubkey, kind, d) = coordinate; let versions = by_key .get(&(pubkey.as_str(), *kind, d.as_str())) .cloned() @@ -600,12 +657,33 @@ async fn validate_addressable(ctx: &Ctx<'_>) -> Result { report.skipped_after_cutoff += 1; continue; } - let events: Vec = versions.iter().map(|r| r.event()).collect(); - let expected = oracle::replaceable_winner(&events).map(|e| e.id.clone()); - let compact_id = compact::fetch_replaceable(ctx.compact, *kind, pubkey, d) .await? .map(|e| e.id); + let legacy_id = if legacy_checked < o.legacy_sample { + legacy_checked += 1; + Some( + query::fetch_replaceable(ctx.legacy, *kind, pubkey, d) + .await? + .map(|e| e.id), + ) + } else { + None + }; + reads.push((coordinate, versions, compact_id, legacy_id)); + } + let changed = coordinates_changed_after(ctx.source, db, batch, ctx.cutoff).await?; + + for (coordinate, versions, compact_id, legacy_id) in reads { + if changed.contains(coordinate) { + report.skipped_after_cutoff += 1; + continue; + } + let (pubkey, kind, d) = coordinate; + let label = format!("{kind}:{pubkey}:{d}"); + let events: Vec = versions.iter().map(|r| r.event()).collect(); + let expected = oracle::replaceable_winner(&events).map(|e| e.id.clone()); + let outcome = if compact_id == expected { Outcome::Match } else { @@ -619,13 +697,9 @@ async fn validate_addressable(ctx: &Ctx<'_>) -> Result { o.max_examples, ); - if legacy_checked >= o.legacy_sample { + let Some(legacy_id) = legacy_id else { continue; - } - legacy_checked += 1; - let legacy_id = query::fetch_replaceable(ctx.legacy, *kind, pubkey, d) - .await? - .map(|e| e.id); + }; let outcome = if legacy_id == expected { Outcome::Match } else if oracle::tie_width(&events) > 1 { @@ -650,6 +724,100 @@ async fn validate_addressable(ctx: &Ctx<'_>) -> Result { Ok(report) } +/// Pubkeys with a kind 0 row indexed after `cutoff`. +/// +/// Validation calls this after the compact and legacy reads, so an insert +/// that landed between the source read and those reads is skipped rather +/// than reported as a difference. +pub async fn profiles_changed_after( + source: &Client, + database: &str, + pubkeys: &[String], + cutoff: u32, +) -> Result> { + if pubkeys.is_empty() { + return Ok(HashSet::new()); + } + let mut q = source.query(&format!( + "SELECT DISTINCT pubkey FROM {database}.events_local \ + WHERE kind = 0 AND indexed_at > toDateTime({cutoff}) AND pubkey IN ({})", + placeholders(pubkeys.len()) + )); + for pk in pubkeys { + q = q.bind(pk); + } + let rows = q + .fetch_all::() + .await + .context("re-checking profile freshness")?; + Ok(rows.into_iter().collect()) +} + +/// Zap targets with a receipt indexed after `cutoff`. +pub async fn zap_targets_changed_after( + source: &Client, + database: &str, + targets: &[String], + cutoff: u32, +) -> Result> { + if targets.is_empty() { + return Ok(HashSet::new()); + } + let list = placeholders(targets.len()); + let mut q = source.query(&format!( + "SELECT DISTINCT target FROM ( \ + SELECT tags FROM {database}.events_local \ + WHERE kind = 9735 AND indexed_at > toDateTime({cutoff}) \ + AND hasAny({ZAP_TARGETS_EXPR}, [{list}]) \ + ) ARRAY JOIN {ZAP_TARGETS_EXPR} AS target WHERE target IN ({list})" + )); + for _ in 0..2 { + for target in targets { + q = q.bind(target); + } + } + let rows = q + .fetch_all::() + .await + .context("re-checking zap freshness")?; + Ok(rows.into_iter().collect()) +} + +/// naddr coordinates with a version indexed after `cutoff`. +pub async fn coordinates_changed_after( + source: &Client, + database: &str, + coordinates: &[(String, u16, String)], + cutoff: u32, +) -> Result> { + if coordinates.is_empty() { + return Ok(HashSet::new()); + } + let pubkeys: BTreeSet<&str> = coordinates.iter().map(|(p, _, _)| p.as_str()).collect(); + let kinds: BTreeSet = coordinates.iter().map(|(_, k, _)| *k).collect(); + let kind_list = kinds + .iter() + .map(u16::to_string) + .collect::>() + .join(", "); + let mut q = source.query(&format!( + "SELECT DISTINCT pubkey, kind, {D_TAG_EXPR} AS d FROM {database}.events_local \ + WHERE {} AND kind IN ({kind_list}) AND indexed_at > toDateTime({cutoff}) \ + AND pubkey IN ({})", + Target::Addressable.kind_predicate(), + placeholders(pubkeys.len()) + )); + for pk in &pubkeys { + q = q.bind(*pk); + } + let wanted: HashSet<&(String, u16, String)> = coordinates.iter().collect(); + let rows = q + .fetch_all::<(String, u16, String)>() + .await + .context("re-checking naddr freshness")?; + Ok(rows.into_iter().filter(|c| wanted.contains(c)).collect()) +} + #[cfg(test)] mod tests { use super::*; @@ -672,6 +840,39 @@ mod tests { assert_eq!(report.examples.len(), 2); } + #[test] + fn all_skipped_target_is_inconclusive() { + let mut skipped = TargetReport::new(Target::Zaps, 4); + skipped.skipped_after_cutoff = 4; + assert_eq!(skipped.compared(), 0); + assert!(skipped.inconclusive()); + + let empty = TargetReport::new(Target::Addressable, 0); + assert!(!empty.inconclusive(), "an empty source is vacuously fine"); + + let mut compared = TargetReport::new(Target::Profiles, 2); + compared.skipped_after_cutoff = 1; + compared.record("compact", "k", Outcome::Match, String::new, 5); + assert!(!compared.inconclusive()); + + let report = ValidationReport { + cutoff: 0, + seed: 0, + targets: vec![compared.clone(), skipped], + }; + assert_eq!(report.unexplained(), 0); + assert_eq!(report.inconclusive_targets(), vec![Target::Zaps]); + assert!(!report.passed()); + assert!(report.render().contains("INCONCLUSIVE")); + + let ok = ValidationReport { + cutoff: 0, + seed: 0, + targets: vec![compared], + }; + assert!(ok.passed()); + } + #[test] fn placeholders_format() { assert_eq!(placeholders(3), "?, ?, ?"); diff --git a/crates/pensieve-preview/tests/compact_lookups.rs b/crates/pensieve-preview/tests/compact_lookups.rs index 1b508af..2e00ea7 100644 --- a/crates/pensieve-preview/tests/compact_lookups.rs +++ b/crates/pensieve-preview/tests/compact_lookups.rs @@ -880,7 +880,7 @@ async fn backfill_and_live_ingest_overlap_without_gaps_or_double_counts() { .await .unwrap(); assert_eq!(full.shards_run, p.shards.len()); - backfill::run_gap_pass( + let gap = backfill::run_gap_pass( &db.client, p, dir, @@ -889,6 +889,7 @@ async fn backfill_and_live_ingest_overlap_without_gaps_or_double_counts() { ) .await .unwrap(); + assert_eq!(gap, StopReason::Complete); } assert_matches_oracle(&db, &all, "after repeated backfill (before merges)").await; @@ -1090,6 +1091,7 @@ async fn projection_backfill_validate_and_bench_end_to_end() { .await .unwrap(); assert_eq!(report.unexplained(), 0, "{}", report.render()); + assert!(report.passed(), "{}", report.render()); for t in &report.targets { assert!(t.sampled > 0, "{}: no keys sampled", t.target); } @@ -1144,3 +1146,158 @@ async fn projection_backfill_validate_and_bench_end_to_end() { } db.drop().await; } + +#[tokio::test] +#[ignore = "requires a disposable ClickHouse at PENSIEVE_TEST_CLICKHOUSE_URL"] +async fn gap_pass_honours_runtime_disk_and_backlog_guards() { + let db = TestDb::new().await; + let checkpoint_dir = tempfile::tempdir().unwrap(); + db.insert_sync(&workload(5, 1_600_000_000)).await; + db.apply_migration().await; + + let options = PlanOptions { + shard_by: ShardBy::CreatedAt, + max_rows_per_shard: 1_000, + id_prefix_len: 2, + require_projection: false, + }; + let p = plan::build_plan(&db.client, &db.name, Target::Profiles, &options) + .await + .unwrap(); + let dir = CheckpointDir::new(checkpoint_dir.path(), Target::Profiles); + dir.save_plan(&p, false).unwrap(); + let margin = Duration::from_secs(3600); + + let no_runtime = BackfillOptions { + max_runtime: Some(Duration::ZERO), + ..backfill_options() + }; + let stopped = backfill::run_gap_pass(&db.client, &p, &dir, margin, &no_runtime) + .await + .unwrap(); + assert_eq!(stopped, StopReason::MaxRuntime); + + let no_disk = BackfillOptions { + min_free_bytes: u64::MAX, + ..backfill_options() + }; + let stopped = backfill::run_gap_pass(&db.client, &p, &dir, margin, &no_disk) + .await + .unwrap(); + assert!(matches!(stopped, StopReason::DiskGuard(_)), "{stopped:?}"); + + assert_eq!(db.count("SELECT count() FROM profile_latest").await, 0); + + // A live insert gives the target one part, which exceeds a zero-part limit. + db.insert_sync(&[profile(0x9999, 0x9999, 1, "live")]).await; + let backlog = BackfillOptions { + max_target_parts: 0, + parts_wait_timeout: Duration::ZERO, + ..backfill_options() + }; + let stopped = backfill::run_gap_pass(&db.client, &p, &dir, margin, &backlog) + .await + .unwrap(); + assert!( + matches!(stopped, StopReason::PartsBacklog(_)), + "{stopped:?}" + ); + + assert!( + !dir.gap_pass_path().exists(), + "refused passes must not checkpoint" + ); + db.drop().await; +} + +#[tokio::test] +#[ignore = "requires a disposable ClickHouse at PENSIEVE_TEST_CLICKHOUSE_URL"] +async fn validation_rechecks_keys_changed_during_reads() { + let db = TestDb::new().await; + db.apply_migration().await; + let author = 0x5150; + let target = hex(0x6001); + let quiet_target = hex(0x6002); + db.insert_sync(&[ + profile(0x10, author, 100, "v1"), + profile(0x11, author + 1, 100, "quiet"), + receipt(0x20, &[&["bolt11", "lnbc1u1x"], &["e", &target]]), + receipt(0x21, &[&["bolt11", "lnbc1u1x"], &["e", &quiet_target]]), + event(0x30, author, 100, 30023, "", &[&["d", "slug"]]), + event(0x31, author, 100, 30023, "", &[&["d", "quiet"]]), + ]) + .await; + + // The validator's cutoff and source read happen here... + tokio::time::sleep(Duration::from_millis(1_100)).await; + let cutoff: u32 = db + .client + .query("SELECT toUInt32(now())") + .fetch_one() + .await + .unwrap(); + tokio::time::sleep(Duration::from_millis(1_100)).await; + + // ...then new winners land before the compact reads. + db.insert_sync(&[ + profile(0x12, author, 200, "v2"), + receipt(0x22, &[&["bolt11", "lnbc2u1x"], &["e", &target]]), + event(0x32, author, 200, 30023, "", &[&["d", "slug"]]), + ]) + .await; + + let pubkeys = [hex(author), hex(author + 1)]; + let changed = validate::profiles_changed_after(&db.client, &db.name, &pubkeys, cutoff) + .await + .unwrap(); + assert_eq!(changed.into_iter().collect::>(), vec![hex(author)]); + + let targets = [target.clone(), quiet_target.clone()]; + let changed = validate::zap_targets_changed_after(&db.client, &db.name, &targets, cutoff) + .await + .unwrap(); + assert_eq!(changed.into_iter().collect::>(), vec![target]); + + let coordinates = [ + (hex(author), 30023u16, "slug".to_string()), + (hex(author), 30023u16, "quiet".to_string()), + ]; + let changed = validate::coordinates_changed_after(&db.client, &db.name, &coordinates, cutoff) + .await + .unwrap(); + assert_eq!( + changed.into_iter().collect::>(), + vec![(hex(author), 30023u16, "slug".to_string())] + ); + db.drop().await; +} + +#[tokio::test] +#[ignore = "requires a disposable ClickHouse at PENSIEVE_TEST_CLICKHOUSE_URL"] +async fn validation_with_every_key_skipped_is_inconclusive() { + let db = TestDb::new().await; + db.apply_migration().await; + // Everything was just indexed, so a 5-minute cutoff excludes every key. + db.insert_async(&workload(6, 1_600_000_000)).await; + + let report = validate::run( + &db.client, + &db.name, + &validate::ValidateOptions { + sample: 20, + hot: 5, + sample_permyriad: 10_000, + require_projection: false, + ..validate::ValidateOptions::default() + }, + ) + .await + .unwrap(); + assert_eq!(report.unexplained(), 0); + for t in &report.targets { + assert!(t.sampled > 0 && t.compared() == 0, "{}", report.render()); + } + assert_eq!(report.inconclusive_targets().len(), 3); + assert!(!report.passed(), "{}", report.render()); + db.drop().await; +} diff --git a/docs/preview_compact_lookups.md b/docs/preview_compact_lookups.md index caf9aa6..be2df79 100644 --- a/docs/preview_compact_lookups.md +++ b/docs/preview_compact_lookups.md @@ -174,7 +174,8 @@ Connection settings also come from `CLICKHOUSE_URL`, `CLICKHOUSE_DATABASE`, including inserts carrying old `created_at` values for shards that are already done. Overlap is harmless, as described above. `--gap-pass` adds an optional belt-and-braces re-read of rows with `indexed_at` within - `--gap-margin-mins` of `T0`; it reads the whole target-kind slice. + `--gap-margin-mins` of `T0`; it reads the whole target-kind slice and is + subject to the same runtime, disk and merge-backlog guards as a shard. - **Projection enforcement.** Shards run with `optimize_use_projections=1` and `force_optimize_projection=1`. After each shard the tool also checks `system.query_log`: @@ -183,7 +184,8 @@ Connection settings also come from `CLICKHOUSE_URL`, `CLICKHOUSE_DATABASE`, (plus 1M rows of slack). This catches parts that lack the projection and would silently be scanned - in full. `--allow-full-scan` with `--shard-by id-prefix` is only for rebuilds + in full. A shard with no `query_log` entry cannot be verified, so it is not + checkpointed. The run stops, and re-running redoes that shard. `--allow-full-scan` with `--shard-by id-prefix` is only for rebuilds after the projection is gone; each shard then reads a primary-key range of the whole table. - **Disk guard.** Before each shard, the unreserved space on the table's disk @@ -227,7 +229,12 @@ comparing `status` (bytes on disk per row) with the plan, and keep - Samples keys from **`events_local`**, so keys missing from the compact tables are detected. It uses a hashed random subset plus the hottest keys. - Compares each key at a consistent cutoff: any key with a source row - indexed after `now - --cutoff-margin-secs` is skipped. + indexed after `now - --cutoff-margin-secs` is skipped. Freshness is + checked again after the compact and legacy reads, so a concurrent insert + is skipped rather than reported as a difference. + - A target whose sampled keys were all skipped is reported as + **inconclusive**. The command then exits non-zero, because nothing was + compared. - The compact read must equal the oracle exactly. - Legacy differences are classified as `legacy_tie_break`, `legacy_id_content_mix`, `legacy_duplicate_inflation`, diff --git a/ops/RUNBOOK.md b/ops/RUNBOOK.md index 6183cd6..be07add 100644 --- a/ops/RUNBOOK.md +++ b/ops/RUNBOOK.md @@ -471,7 +471,9 @@ $PL bench --samples 50 --out /home/pensieve/preview-lookups/bench-warm.json $PL bench --samples 50 --drop-caches --include-legacy --legacy-samples 3 --out /home/pensieve/preview-lookups/bench-cold.json ``` -- **`validate`** must report `unexplained differences: 0`. Legacy +- **`validate`** must exit 0: `unexplained differences: 0` and no + `INCONCLUSIVE` targets. A target where every key was skipped by the cutoff + proves nothing, so retry later. Legacy `tie_break`, `id_content_mix`, `duplicate_inflation` and `amount_row_missing` are known legacy defects (see the design doc). - **`bench`** results are what decide the sub-second target. Record p95 and From e4310cecc71735942f79e3f44c55143f79ee7615 Mon Sep 17 00:00:00 2001 From: Jeff Gardner <202880+erskingardner@users.noreply.github.com> Date: Sat, 10 Oct 2026 14:40:44 -0500 Subject: [PATCH 3/6] Bound backfill sessions by an absolute deadline; reject empty validation samples - backfill and gap pass: max_runtime is an absolute session deadline that bounds merge waits and is re-checked immediately before every INSERT, so no INSERT starts after the window closes - validate: reject --sample 0 --hot 0; when sampling finds no keys, check whether the source has any and report the target inconclusive if so - regression tests for a budget expiring mid-wait and for an empty sample of a non-empty source Co-Authored-By: Claude Opus 5.5 --- .../pensieve-preview/src/lookups/backfill.rs | 60 +++++++++--- .../pensieve-preview/src/lookups/validate.rs | 56 +++++++++-- .../pensieve-preview/tests/compact_lookups.rs | 95 +++++++++++++++++++ docs/preview_compact_lookups.md | 13 ++- 4 files changed, 195 insertions(+), 29 deletions(-) diff --git a/crates/pensieve-preview/src/lookups/backfill.rs b/crates/pensieve-preview/src/lookups/backfill.rs index 83f00a6..94e7f53 100644 --- a/crates/pensieve-preview/src/lookups/backfill.rs +++ b/crates/pensieve-preview/src/lookups/backfill.rs @@ -22,7 +22,9 @@ //! `max_memory_usage`, external aggregation and `max_execution_time` set by //! the caller's client. Before each shard the free space on the table's disk //! must cover the remaining estimate plus a reserve, and the target must not -//! have too many unmerged parts. `max_runtime` and `max_shards` stop between +//! have too many unmerged parts. `max_runtime` is an absolute session deadline: +//! it bounds merge waits and is re-checked immediately before every INSERT, so +//! no INSERT starts after the window closes. `max_shards` stops between //! shards; completed shards are checkpointed so a re-run resumes. use std::time::{Duration, Instant}; @@ -165,6 +167,7 @@ pub async fn run_backfill( } let started = Instant::now(); + let deadline = session_deadline(options); let done: std::collections::BTreeSet = checkpoints .completed_shards()? .into_iter() @@ -192,9 +195,7 @@ pub async fn run_backfill( continue; } - if let Some(max) = options.max_runtime - && started.elapsed() >= max - { + if expired(deadline) { return Ok(report( total, shards_done, @@ -230,13 +231,17 @@ pub async fn run_backfill( StopReason::DiskGuard(refusal), )); } - if let Some(backlog) = wait_for_merges(client, plan, options).await? { + if let Some(stop) = wait_for_merges(client, plan, options, deadline).await? { + return Ok(report(total, shards_done, shards_run, rows_written, stop)); + } + // Guards may have taken a while; never start an INSERT past the deadline. + if expired(deadline) { return Ok(report( total, shards_done, shards_run, rows_written, - StopReason::PartsBacklog(backlog), + StopReason::MaxRuntime, )); } @@ -299,7 +304,8 @@ pub async fn run_gap_pass( margin: Duration, options: &BackfillOptions, ) -> Result { - if options.max_runtime.is_some_and(|m| m.is_zero()) { + let deadline = session_deadline(options); + if expired(deadline) { return Ok(StopReason::MaxRuntime); } let margin = margin.as_secs() as u32; @@ -320,11 +326,17 @@ pub async fn run_gap_pass( .fetch_one::() .await .with_context(|| format!("sizing {} gap-closure pass", plan.target))?; + if expired(deadline) { + return Ok(StopReason::MaxRuntime); + } if let Some(refusal) = disk_guard(client, plan, estimated_rows, options).await? { return Ok(StopReason::DiskGuard(refusal)); } - if let Some(backlog) = wait_for_merges(client, plan, options).await? { - return Ok(StopReason::PartsBacklog(backlog)); + if let Some(stop) = wait_for_merges(client, plan, options, deadline).await? { + return Ok(stop); + } + if expired(deadline) { + return Ok(StopReason::MaxRuntime); } // Read amplification is expected here (indexed_at is not a key column). @@ -479,24 +491,41 @@ async fn disk_guard( Ok(None) } +/// Absolute session deadline for `max_runtime`, fixed when a run starts. +fn session_deadline(options: &BackfillOptions) -> Option { + options.max_runtime.map(|max| Instant::now() + max) +} + +fn expired(deadline: Option) -> bool { + deadline.is_some_and(|d| Instant::now() >= d) +} + +/// Wait while the target has too many active parts. Stops with +/// `PartsBacklog` after `parts_wait_timeout`, or `MaxRuntime` when the +/// session deadline passes first. async fn wait_for_merges( client: &Client, plan: &Plan, options: &BackfillOptions, -) -> Result> { - let deadline = Instant::now() + options.parts_wait_timeout; + session: Option, +) -> Result> { + let parts_deadline = Instant::now() + options.parts_wait_timeout; loop { let parts = ops::table_parts(client, &plan.database, plan.target.table()).await?; if parts.parts <= options.max_target_parts { return Ok(None); } - if Instant::now() >= deadline { - return Ok(Some(format!( + if expired(session) { + return Ok(Some(StopReason::MaxRuntime)); + } + let now = Instant::now(); + if now >= parts_deadline { + return Ok(Some(StopReason::PartsBacklog(format!( "{} still has {} active parts (limit {})", plan.target.table(), parts.parts, options.max_target_parts - ))); + )))); } tracing::info!( table = plan.target.table(), @@ -504,7 +533,8 @@ async fn wait_for_merges( limit = options.max_target_parts, "waiting for merges" ); - tokio::time::sleep(PARTS_POLL).await; + let wake = session.map_or(parts_deadline, |d| d.min(parts_deadline)); + tokio::time::sleep(PARTS_POLL.min(wake.saturating_duration_since(now))).await; } } diff --git a/crates/pensieve-preview/src/lookups/validate.rs b/crates/pensieve-preview/src/lookups/validate.rs index 673f5d8..55a5a56 100644 --- a/crates/pensieve-preview/src/lookups/validate.rs +++ b/crates/pensieve-preview/src/lookups/validate.rs @@ -15,13 +15,14 @@ //! defects (timestamp tie-breaks, id/content from different rows, //! duplicate inflation, missing amount rows) or unexplained. //! -//! A target whose sampled keys were all skipped is **inconclusive**, not a -//! pass. +//! A target with no compared keys is **inconclusive**, not a pass: either +//! every sampled key was skipped, or sampling found no keys although the +//! source has some. use std::collections::{BTreeMap, BTreeSet, HashMap, HashSet}; use std::time::Duration; -use anyhow::{Context, Result}; +use anyhow::{Context, Result, bail}; use clickhouse::Client; use serde::{Deserialize, Serialize}; @@ -113,6 +114,8 @@ pub struct TargetReport { pub target: Target, /// Keys sampled. pub sampled: usize, + /// Whether `events_local` has any source key for this target. + pub source_has_keys: bool, /// Keys skipped because source rows were indexed after the cutoff /// (before or during validation). pub skipped_after_cutoff: usize, @@ -129,6 +132,7 @@ impl TargetReport { Self { target, sampled, + source_has_keys: sampled > 0, skipped_after_cutoff: 0, compact: BTreeMap::new(), legacy: BTreeMap::new(), @@ -165,10 +169,11 @@ impl TargetReport { self.compact.values().sum() } - /// Keys were sampled but none could be compared, so the target provides - /// no correctness evidence. + /// Nothing was compared although the source has keys, so the target + /// provides no correctness evidence. Only a truly empty source passes + /// with zero comparisons. pub fn inconclusive(&self) -> bool { - self.sampled > 0 && self.compared() == 0 + self.compared() == 0 && (self.sampled > 0 || self.source_has_keys) } /// Unexplained differences on either side. @@ -234,8 +239,9 @@ impl ValidationReport { let inconclusive = self.inconclusive_targets(); if !inconclusive.is_empty() { out.push_str(&format!( - "INCONCLUSIVE (no keys compared): {inconclusive:?}; lower --cutoff-margin-secs \ - only if ingest is quiet, or retry later\n" + "INCONCLUSIVE (source has keys but none were compared): {inconclusive:?}; \ + widen --sample/--hot/--sample-permyriad, or retry later if keys were \ + skipped by the cutoff\n" )); } out @@ -272,6 +278,9 @@ pub async fn run( database: &str, options: &ValidateOptions, ) -> Result { + if options.sample == 0 && options.hot == 0 { + bail!("validation needs --sample or --hot above 0"); + } let now = ops::server_now(client).await?; let cutoff = now.saturating_sub(options.cutoff_margin.as_secs() as u32); let source = projection_client(client, options.require_projection); @@ -286,11 +295,14 @@ pub async fn run( let mut targets = Vec::new(); for target in &options.targets { - let report = match target { + let mut report = match target { Target::Profiles => validate_profiles(&ctx).await?, Target::Zaps => validate_zaps(&ctx).await?, Target::Addressable => validate_addressable(&ctx).await?, }; + if report.sampled == 0 { + report.source_has_keys = source_has_keys(&ctx, *target).await?; + } targets.push(report); } Ok(ValidationReport { @@ -309,6 +321,25 @@ struct Ctx<'a> { cutoff: u32, } +/// Whether the source has at least one key for `target` (used when sampling +/// found none, to tell an empty source from an empty sample). +async fn source_has_keys(ctx: &Ctx<'_>, target: Target) -> Result { + let db = ctx.db; + let filter = match target { + Target::Zaps => format!("kind = 9735 AND notEmpty({ZAP_TARGETS_EXPR})"), + other => other.kind_predicate().to_string(), + }; + let found = ctx + .source + .query(&format!( + "SELECT count() FROM (SELECT 1 FROM {db}.events_local WHERE {filter} LIMIT 1)" + )) + .fetch_one::() + .await + .with_context(|| format!("checking {target} source keys"))?; + Ok(found > 0) +} + fn placeholders(n: usize) -> String { vec!["?"; n].join(", ") } @@ -850,6 +881,13 @@ mod tests { let empty = TargetReport::new(Target::Addressable, 0); assert!(!empty.inconclusive(), "an empty source is vacuously fine"); + let mut empty_sample = TargetReport::new(Target::Addressable, 0); + empty_sample.source_has_keys = true; + assert!( + empty_sample.inconclusive(), + "an empty sample of a non-empty source proves nothing" + ); + let mut compared = TargetReport::new(Target::Profiles, 2); compared.skipped_after_cutoff = 1; compared.record("compact", "k", Outcome::Match, String::new, 5); diff --git a/crates/pensieve-preview/tests/compact_lookups.rs b/crates/pensieve-preview/tests/compact_lookups.rs index 2e00ea7..750f488 100644 --- a/crates/pensieve-preview/tests/compact_lookups.rs +++ b/crates/pensieve-preview/tests/compact_lookups.rs @@ -1301,3 +1301,98 @@ async fn validation_with_every_key_skipped_is_inconclusive() { assert!(!report.passed(), "{}", report.render()); db.drop().await; } + +#[tokio::test] +#[ignore = "requires a disposable ClickHouse at PENSIEVE_TEST_CLICKHOUSE_URL"] +async fn validation_with_empty_sample_of_nonempty_source_is_inconclusive() { + let db = TestDb::new().await; + db.insert_sync(&workload(7, 1_600_000_000)).await; + db.apply_migration().await; + + // Sampling disabled entirely is rejected. + let none = validate::ValidateOptions { + sample: 0, + hot: 0, + require_projection: false, + ..validate::ValidateOptions::default() + }; + assert!(validate::run(&db.client, &db.name, &none).await.is_err()); + + // A random-only sample that selects no keys cannot pass on a non-empty + // source, even though the compact tables were never backfilled. + let empty_fraction = validate::ValidateOptions { + sample: 20, + hot: 0, + sample_permyriad: 0, + cutoff_margin: Duration::ZERO, + require_projection: false, + ..validate::ValidateOptions::default() + }; + let report = validate::run(&db.client, &db.name, &empty_fraction) + .await + .unwrap(); + for t in &report.targets { + assert_eq!(t.sampled, 0); + assert!(t.source_has_keys, "{}", report.render()); + } + assert_eq!(report.inconclusive_targets().len(), 3); + assert!(!report.passed(), "{}", report.render()); + db.drop().await; +} + +#[tokio::test] +#[ignore = "requires a disposable ClickHouse at PENSIEVE_TEST_CLICKHOUSE_URL"] +async fn session_deadline_expiring_during_merge_wait_starts_no_insert() { + let db = TestDb::new().await; + let checkpoint_dir = tempfile::tempdir().unwrap(); + db.insert_sync(&workload(8, 1_600_000_000)).await; + db.apply_migration().await; + // One live insert gives the target a part, so a zero-part limit waits. + db.insert_sync(&[profile(0x9998, 0x9998, 1, "live")]).await; + + let options = PlanOptions { + shard_by: ShardBy::CreatedAt, + max_rows_per_shard: 1_000, + id_prefix_len: 2, + require_projection: false, + }; + let p = plan::build_plan(&db.client, &db.name, Target::Profiles, &options) + .await + .unwrap(); + let dir = CheckpointDir::new(checkpoint_dir.path(), Target::Profiles); + dir.save_plan(&p, false).unwrap(); + let rows_before = db.count("SELECT count() FROM profile_latest").await; + + // A positive budget that runs out while waiting for merges (the parts + // wait alone would allow an hour). + let waiting = BackfillOptions { + max_runtime: Some(Duration::from_secs(2)), + max_target_parts: 0, + parts_wait_timeout: Duration::from_secs(3600), + ..backfill_options() + }; + let started = std::time::Instant::now(); + let report = backfill::run_backfill(&db.client, &p, &dir, &waiting) + .await + .unwrap(); + assert_eq!(report.stopped, StopReason::MaxRuntime); + assert_eq!(report.shards_run, 0); + assert!( + started.elapsed() < Duration::from_secs(20), + "wait was not bounded by the deadline" + ); + + let stopped = backfill::run_gap_pass(&db.client, &p, &dir, Duration::from_secs(3600), &waiting) + .await + .unwrap(); + assert_eq!(stopped, StopReason::MaxRuntime); + + assert!(dir.completed_shards().unwrap().is_empty()); + assert!(!dir.gap_pass_path().exists()); + assert_eq!( + db.count("SELECT count() FROM profile_latest").await, + rows_before, + "no INSERT may start after the deadline" + ); + db.drop().await; +} diff --git a/docs/preview_compact_lookups.md b/docs/preview_compact_lookups.md index be2df79..e9d364f 100644 --- a/docs/preview_compact_lookups.md +++ b/docs/preview_compact_lookups.md @@ -194,8 +194,10 @@ Connection settings also come from `CLICKHOUSE_URL`, `CLICKHOUSE_DATABASE`, - **Merge backlog.** While the target has more than `--max-target-parts` active parts (default 300), the tool waits for merges. After 30 minutes it stops cleanly. -- **Runtime.** `--max-runtime-mins`, `--max-shards` and `--delay-secs` bound - each session. Each finished shard writes a JSON checkpoint containing its +- **Runtime.** `--max-runtime-mins` is an absolute session deadline. It also + bounds merge waits and is re-checked right before every INSERT, so no + INSERT starts after the window closes. `--max-shards` and `--delay-secs` + further bound each session. Each finished shard writes a JSON checkpoint containing its `system.query_log` entry. - **Progress.** Each shard logs done/total, percent, rows read and written, duration and ETA. @@ -232,9 +234,10 @@ comparing `status` (bytes on disk per row) with the plan, and keep indexed after `now - --cutoff-margin-secs` is skipped. Freshness is checked again after the compact and legacy reads, so a concurrent insert is skipped rather than reported as a difference. - - A target whose sampled keys were all skipped is reported as - **inconclusive**. The command then exits non-zero, because nothing was - compared. + - A target is reported as **inconclusive**, and the command exits non-zero, + when nothing was compared but the source has keys. That covers every + sampled key being skipped, and sampling selecting no keys. Disabling + sampling (`--sample 0 --hot 0`) is rejected. - The compact read must equal the oracle exactly. - Legacy differences are classified as `legacy_tie_break`, `legacy_id_content_mix`, `legacy_duplicate_inflation`, From 74c511ea3d2076b135a56e75ea113d4073f25e25 Mon Sep 17 00:00:00 2001 From: Jeff Gardner <202880+erskingardner@users.noreply.github.com> Date: Sat, 10 Oct 2026 15:01:20 -0500 Subject: [PATCH 4/6] Add a projections-off rehearsal setting to pensieve-preview PREVIEW_DISABLE_PROJECTIONS=true runs every preview query with optimize_use_projections=0, so real nstr.to traffic can prove the compact lookups before the events_by_kind projection is dropped. Queries are capped at 10 s and cancelled when their request is dropped, so any lookup still on its legacy path fails fast instead of piling up table scans; startup warns unless every lookup is compact. Off by default; rollback is a restart. - integration test asserts every preview query logs the setting, a cap and no projection use (and fails when the setting is off) - runbook step 9: rehearsal procedure, pass criteria and rollback Co-Authored-By: Claude Opus 5.5 --- crates/pensieve-preview/src/config.rs | 54 ++++++ crates/pensieve-preview/src/state.rs | 31 +++- .../pensieve-preview/tests/compact_lookups.rs | 167 ++++++++++++++++++ docs/preview_compact_lookups.md | 16 ++ env.local.example | 2 + env.production.example | 4 + ops/RUNBOOK.md | 22 +++ 7 files changed, 293 insertions(+), 3 deletions(-) diff --git a/crates/pensieve-preview/src/config.rs b/crates/pensieve-preview/src/config.rs index 1a387fd..6c8f834 100644 --- a/crates/pensieve-preview/src/config.rs +++ b/crates/pensieve-preview/src/config.rs @@ -35,6 +35,11 @@ pub struct Config { /// Which lookups read the compact ClickHouse tables (migration 017) /// instead of `events_local`. Defaults to none (legacy reads). pub compact_lookups: CompactLookups, + + /// Run every preview query with `optimize_use_projections = 0`, a + /// rehearsal of production without the `events_by_kind` projection. + /// Defaults to false. + pub disable_projections: bool, } impl Config { @@ -53,6 +58,9 @@ impl Config { /// - `PREVIEW_VIP_FILE`: Path to VIP pubkeys file (default: ".vip") /// - `PREVIEW_COMPACT_LOOKUPS`: `none` (default), `all`, or a comma list of /// `profiles`, `zaps`, `addressable`. An unrecognised value is an error. + /// - `PREVIEW_DISABLE_PROJECTIONS`: `true`/`false` (default `false`). Runs + /// every preview query with projections disabled and bounded; see + /// [`crate::state::PROJECTIONS_OFF_SETTINGS`]. pub fn from_env() -> anyhow::Result { let bind_addr = std::env::var("PREVIEW_BIND_ADDR").unwrap_or_else(|_| "0.0.0.0:8081".to_string()); @@ -79,6 +87,18 @@ impl Config { .parse::() .context("PREVIEW_COMPACT_LOOKUPS")?; + let disable_projections = parse_bool( + "PREVIEW_DISABLE_PROJECTIONS", + &std::env::var("PREVIEW_DISABLE_PROJECTIONS").unwrap_or_default(), + )?; + if disable_projections && compact_lookups != CompactLookups::ALL { + tracing::warn!( + compact_lookups = %compact_lookups, + "PREVIEW_DISABLE_PROJECTIONS is set while some lookups still use legacy reads; \ + those will scan events_local and hit the query time cap" + ); + } + tracing::info!( bind_addr = %bind_addr, clickhouse_url = %clickhouse_url, @@ -87,6 +107,7 @@ impl Config { vip_count = vip_pubkeys.len(), vip_file = %vip_file, compact_lookups = %compact_lookups, + disable_projections, "preview configuration loaded" ); @@ -98,10 +119,21 @@ impl Config { site_name, vip_pubkeys: Arc::new(vip_pubkeys), compact_lookups, + disable_projections, }) } } +/// Parse a boolean environment value (`1/true/yes/on`, `0/false/no/off`, +/// empty = false). +fn parse_bool(name: &str, value: &str) -> anyhow::Result { + match value.trim().to_ascii_lowercase().as_str() { + "" | "0" | "false" | "no" | "off" => Ok(false), + "1" | "true" | "yes" | "on" => Ok(true), + other => anyhow::bail!("{name}: expected true or false, got {other:?}"), + } +} + /// Load VIP pubkeys from a file. /// /// The file should contain one pubkey per line. Supported formats: @@ -186,6 +218,7 @@ mod tests { "PREVIEW_SITE_NAME", "PREVIEW_VIP_FILE", "PREVIEW_COMPACT_LOOKUPS", + "PREVIEW_DISABLE_PROJECTIONS", ]; /// Helper to run config tests with isolated env vars. @@ -231,6 +264,27 @@ mod tests { assert_eq!(config.site_name, "Nostr Preview"); assert!(config.vip_pubkeys.is_empty()); assert_eq!(config.compact_lookups, CompactLookups::NONE); + assert!(!config.disable_projections); + }); + } + + #[test] + fn config_disable_projections_parsed() { + for (value, expected) in [ + ("true", true), + ("1", true), + ("ON", true), + ("false", false), + ("0", false), + ("", false), + ] { + with_env_vars(&[("PREVIEW_DISABLE_PROJECTIONS", value)], || { + let config = Config::from_env().unwrap(); + assert_eq!(config.disable_projections, expected, "{value:?}"); + }); + } + with_env_vars(&[("PREVIEW_DISABLE_PROJECTIONS", "maybe")], || { + assert!(Config::from_env().is_err()); }); } diff --git a/crates/pensieve-preview/src/state.rs b/crates/pensieve-preview/src/state.rs index ba2c3d7..12462ed 100644 --- a/crates/pensieve-preview/src/state.rs +++ b/crates/pensieve-preview/src/state.rs @@ -56,12 +56,37 @@ pub struct AppState { pub og_cache: OgImageCache, } +/// ClickHouse settings applied to every preview query when +/// `PREVIEW_DISABLE_PROJECTIONS` is set. +/// +/// Besides disabling projections, they bound any query that would now scan +/// `events_local` (a lookup still on its legacy path): it fails after 10 +/// seconds, matching the HTTP request timeout, and is cancelled if the request +/// is dropped first, so a rehearsal cannot pile up long-running scans. +pub const PROJECTIONS_OFF_SETTINGS: [(&str, &str); 4] = [ + ("optimize_use_projections", "0"), + ("max_execution_time", "10"), + ("timeout_overflow_mode", "throw"), + ("cancel_http_readonly_queries_on_client_close", "1"), +]; + +/// Build the preview's ClickHouse client from configuration. +pub fn clickhouse_client(config: &Config) -> Client { + let mut client = Client::default() + .with_url(&config.clickhouse_url) + .with_database(&config.clickhouse_database); + if config.disable_projections { + for (name, value) in PROJECTIONS_OFF_SETTINGS { + client = client.with_option(name, value); + } + } + client +} + impl AppState { /// Create a new application state from configuration. pub fn new(config: Config) -> Self { - let clickhouse = Client::default() - .with_url(&config.clickhouse_url) - .with_database(&config.clickhouse_database); + let clickhouse = clickhouse_client(&config); let cache = Cache::builder() .max_capacity(DEFAULT_CACHE_CAPACITY) diff --git a/crates/pensieve-preview/tests/compact_lookups.rs b/crates/pensieve-preview/tests/compact_lookups.rs index 750f488..e58e33a 100644 --- a/crates/pensieve-preview/tests/compact_lookups.rs +++ b/crates/pensieve-preview/tests/compact_lookups.rs @@ -1396,3 +1396,170 @@ async fn session_deadline_expiring_during_merge_wait_starts_no_insert() { ); db.drop().await; } + +#[tokio::test] +#[ignore = "requires a disposable ClickHouse at PENSIEVE_TEST_CLICKHOUSE_URL"] +async fn projections_off_rehearsal_applies_to_every_preview_query() { + use pensieve_preview::{AppState, Config}; + + let db = TestDb::new().await; + let checkpoint_dir = tempfile::tempdir().unwrap(); + insert_bulk(&db, 200_000).await; + db.apply_migration().await; + for target in Target::ALL { + let options = PlanOptions { + shard_by: ShardBy::CreatedAt, + max_rows_per_shard: 50_000, + id_prefix_len: 2, + require_projection: false, + }; + let p = plan::build_plan(&db.client, &db.name, target, &options) + .await + .unwrap(); + let dir = CheckpointDir::new(checkpoint_dir.path(), target); + dir.save_plan(&p, false).unwrap(); + let report = backfill::run_backfill(&db.client, &p, &dir, &backfill_options()) + .await + .unwrap(); + assert_eq!(report.stopped, StopReason::Complete); + } + + let state_for = |compact_lookups| { + AppState::new(Config { + bind_addr: String::new(), + clickhouse_url: std::env::var("PENSIEVE_TEST_CLICKHOUSE_URL").unwrap(), + clickhouse_database: db.name.clone(), + base_url: String::new(), + site_name: String::new(), + vip_pubkeys: Default::default(), + compact_lookups, + disable_projections: true, + }) + }; + + let pubkey: String = db + .client + .query("SELECT pubkey FROM profile_latest LIMIT 1") + .fetch_one() + .await + .unwrap(); + let zap_target: String = db + .client + .query("SELECT target_event_id FROM zap_receipts_by_target LIMIT 1") + .fetch_one() + .await + .unwrap(); + let (author, kind, d): (String, u16, String) = db + .client + .query("SELECT pubkey, kind, d_tag FROM addressable_latest LIMIT 1") + .fetch_one() + .await + .unwrap(); + + // Full rehearsal: every lookup on the compact path, projections off. + let state = state_for(CompactLookups::ALL); + assert!( + state + .lookups + .fetch_profile(&pubkey) + .await + .unwrap() + .is_some() + ); + assert_eq!( + state + .lookups + .fetch_profiles(std::slice::from_ref(&pubkey)) + .await + .unwrap() + .len(), + 1 + ); + state + .lookups + .fetch_display_names(std::slice::from_ref(&pubkey)) + .await + .unwrap(); + // Engagement also reads the reaction/comment/repost tables, which this + // fixture lacks; zap totals are covered by the compact zap read. + let naddr = state + .lookups + .fetch_replaceable(kind, &author, &d) + .await + .unwrap() + .expect("naddr row"); + assert!( + state + .lookups + .fetch_event_by_id(&naddr.id) + .await + .unwrap() + .is_some() + ); + let zaps = compact::fetch_zap_totals(&state.clickhouse, &zap_target) + .await + .unwrap(); + assert!(zaps.zap_count > 0); + + // A legacy lookup under the rehearsal is bounded rather than scanning + // with the projection. + let legacy = state_for(CompactLookups::NONE); + legacy.lookups.fetch_profile(&pubkey).await.unwrap(); + + #[derive(Debug, serde::Deserialize, clickhouse::Row)] + struct LoggedQuery { + log_comment: String, + projections: Vec, + use_projections: String, + max_time: String, + } + db.exec("SYSTEM FLUSH LOGS").await; + let rows = db + .admin + .query( + "SELECT log_comment, projections, \ + Settings['optimize_use_projections'] AS use_projections, \ + Settings['max_execution_time'] AS max_time \ + FROM system.query_log \ + WHERE event_date >= yesterday() AND type = 'QueryFinish' \ + AND current_database = ? \ + AND (startsWith(log_comment, 'pensieve-preview:') \ + OR query LIKE '%FROM zap_receipts_by_target%WHERE target_event_id%')", + ) + .bind(&db.name) + .fetch_all::() + .await + .unwrap(); + let comments: Vec<&str> = rows.iter().map(|r| r.log_comment.as_str()).collect(); + for expected in [ + "pensieve-preview:profile:compact", + "pensieve-preview:profiles:compact", + "pensieve-preview:display_names:compact", + "pensieve-preview:replaceable:compact", + "pensieve-preview:event:legacy", + "pensieve-preview:profile:legacy", + ] { + assert!( + comments.contains(&expected), + "missing {expected}: {comments:?}" + ); + } + assert!( + rows.iter().any(|r| r.log_comment.is_empty()), + "compact zap read not logged" + ); + for r in &rows { + let comment = &r.log_comment; + assert_eq!( + r.use_projections, "0", + "{comment} ran with projections enabled" + ); + assert!( + r.projections.is_empty(), + "{comment} used {:?}", + r.projections + ); + assert!(!r.max_time.is_empty(), "{comment} has no execution cap"); + } + db.drop().await; +} diff --git a/docs/preview_compact_lookups.md b/docs/preview_compact_lookups.md index e9d364f..df72a56 100644 --- a/docs/preview_compact_lookups.md +++ b/docs/preview_compact_lookups.md @@ -261,6 +261,22 @@ Preview queries carry `log_comment = 'pensieve-preview:: so `system.query_log` gives production latency per lookup and source during rollout. +### Projections-off rehearsal + +`PREVIEW_DISABLE_PROJECTIONS=true` runs **every** preview query with +`optimize_use_projections = 0`, which is how the preview will behave once the +projection is gone. It lets real nstr.to traffic prove the change before the +irreversible drop. To keep the rehearsal from piling up scans, each query is +also capped: +- `max_execution_time = 10` with `timeout_overflow_mode = throw`, matching the + HTTP request timeout; +- `cancel_http_readonly_queries_on_client_close = 1`, so a query whose request + was dropped is cancelled. + +Any lookup still on its legacy path fails fast instead of scanning for a +minute, and startup logs a warning unless `PREVIEW_COMPACT_LOOKUPS=all`. Turn +it off with `PREVIEW_DISABLE_PROJECTIONS=false` and a restart. + ## Requires production validation The checks below can only be done on the production server. Locally, the diff --git a/env.local.example b/env.local.example index bec1801..88658f8 100644 --- a/env.local.example +++ b/env.local.example @@ -30,3 +30,5 @@ PREVIEW_BASE_URL=http://localhost:8081 PREVIEW_SITE_NAME=Nostr Preview # Compact ClickHouse lookups (migration 017): none, all, or profiles,zaps,addressable # PREVIEW_COMPACT_LOOKUPS=none +# Rehearse without projections (optimize_use_projections=0 on every preview query) +# PREVIEW_DISABLE_PROJECTIONS=false diff --git a/env.production.example b/env.production.example index 7ac672c..5ce779d 100644 --- a/env.production.example +++ b/env.production.example @@ -53,6 +53,10 @@ PREVIEW_SITE_NAME=Nostr Preview # profiles,zaps,addressable. Reversible with a restart. See # docs/preview_compact_lookups.md before enabling. # PREVIEW_COMPACT_LOOKUPS=none +# Rehearse running without the events_by_kind projection: every preview query +# runs with optimize_use_projections=0 and a 10 s cap. Use only after +# PREVIEW_COMPACT_LOOKUPS=all is validated (see ops/RUNBOOK.md). +# PREVIEW_DISABLE_PROJECTIONS=false # ═══════════════════════════════════════════════════════════════════════════ # Ingester diff --git a/ops/RUNBOOK.md b/ops/RUNBOOK.md index be07add..cae344b 100644 --- a/ops/RUNBOOK.md +++ b/ops/RUNBOOK.md @@ -534,6 +534,27 @@ For each consumer listed, measure it with deciding. The dashboard queries were previously measured at about 9.2 s for 14 queries without the projection. +### 9. Projections-off rehearsal (before any removal) + +After the preview has run on `PREVIEW_COMPACT_LOOKUPS=all` without issues: + +1. Set `PREVIEW_DISABLE_PROJECTIONS=true` in `/etc/pensieve/pensieve.env` and + restart `pensieve-preview`. Every preview query then runs with + `optimize_use_projections = 0`, capped at 10 s and cancelled when its + request is dropped. +2. Run it for at least a day of normal traffic, watching the step 6 latency + query, `slow preview lookup` warnings and the preview error rate. This + query must show the setting applied and no projection used: + + ```bash + just ch-query "SELECT log_comment, Settings['optimize_use_projections'] AS proj, count() n, quantile(0.95)(query_duration_ms) p95, max(query_duration_ms) max, countIf(notEmpty(projections)) used_projection FROM system.query_log WHERE event_time > now() - INTERVAL 1 DAY AND type = 'QueryFinish' AND startsWith(log_comment, 'pensieve-preview:') GROUP BY log_comment, proj ORDER BY log_comment" + just ch-query "SELECT log_comment, count() FROM system.query_log WHERE event_time > now() - INTERVAL 1 DAY AND type != 'QueryFinish' AND type != 'QueryStart' AND startsWith(log_comment, 'pensieve-preview:') GROUP BY log_comment" + ``` + + Pass: `proj = 0` and `used_projection = 0` everywhere, p95 well under 1 s + for the compact lookups, and no rise in failed queries (second query). +3. **Rollback:** set `PREVIEW_DISABLE_PROJECTIONS=false` and restart. + ### Projection removal (separate, manual) **Not part of this rollout.** Only consider it after all of these hold: @@ -541,6 +562,7 @@ deciding. The dashboard queries were previously measured at about 9.2 s for - `validate` is clean; - `bench` meets the target; - the preview has run on `all` for an agreed period; +- the projections-off rehearsal (step 9) passed; - the consumer audit above is clear or each consumer is accepted; - the removal is written up as its own prepared migration, following the `prepared/016` pattern (preflight, asynchronous `DROP PROJECTION`, rollback From b575f5b2748575bc69914161c32d9ff93922aedd Mon Sep 17 00:00:00 2001 From: Jeff Gardner <202880+erskingardner@users.noreply.github.com> Date: Sat, 10 Oct 2026 15:06:39 -0500 Subject: [PATCH 5/6] Correct ingest-safety docs to match production's synchronous inserts Production preflight (2026-10-10) shows the ingester inserts synchronously: async_insert = 0 for its user and system.asynchronous_insert_log is empty, because the async settings in ops/production/clickhouse/config.xml are profile settings in a server config.d file, which ClickHouse does not apply. A failing view therefore fails the INSERT (retried and reindex-queued) rather than dropping rows silently. - design doc and migration 017 header describe the real failure mode - runbook preflight and post-migration checks use query_views_log and query_log insert exceptions instead of the empty asynchronous_insert_log - malformed-event test covers synchronous as well as async inserts Co-Authored-By: Claude Opus 5.5 --- .../pensieve-preview/tests/compact_lookups.rs | 3 +++ .../prepared/017_preview_compact_lookups.sql | 4 +-- docs/preview_compact_lookups.md | 26 ++++++++++++++----- ops/RUNBOOK.md | 22 +++++++++++----- 4 files changed, 39 insertions(+), 16 deletions(-) diff --git a/crates/pensieve-preview/tests/compact_lookups.rs b/crates/pensieve-preview/tests/compact_lookups.rs index e58e33a..d582b8e 100644 --- a/crates/pensieve-preview/tests/compact_lookups.rs +++ b/crates/pensieve-preview/tests/compact_lookups.rs @@ -736,6 +736,9 @@ async fn async_insert_flush_survives_malformed_events() { event(9, 3, 5, 10002, "", &[&["d", "\u{1F600}"]]), event(10, 3, 5, 39999, "", &[&["d", ""], &["d", "second"]]), ]; + // Production inserts synchronously; also cover fire-and-forget async + // inserts, where a throwing view would drop rows silently. + db.insert_sync(&malformed).await; db.insert_async(&malformed).await; db.exec("SYSTEM FLUSH LOGS").await; diff --git a/docs/migrations/prepared/017_preview_compact_lookups.sql b/docs/migrations/prepared/017_preview_compact_lookups.sql index 17e1b07..dc75c8b 100644 --- a/docs/migrations/prepared/017_preview_compact_lookups.sql +++ b/docs/migrations/prepared/017_preview_compact_lookups.sql @@ -22,8 +22,8 @@ -- - Follow docs/preview_compact_lookups.md and the ops/RUNBOOK.md section -- "Preview compact lookups" for preflight, backfill, validation and rollout. -- - Every expression below must be non-throwing for arbitrary tag arrays. A --- throwing materialized view fails the whole async-insert flush, and with --- wait_for_async_insert=0 the ingester never sees that error. +-- throwing materialized view fails the whole events_local insert (with +-- async inserts and wait_for_async_insert=0 it would drop rows silently). -- -- Correctness model (see docs/preview_compact_lookups.md): -- Reads never depend on background merges. Every read collapses duplicate diff --git a/docs/preview_compact_lookups.md b/docs/preview_compact_lookups.md index df72a56..37d47af 100644 --- a/docs/preview_compact_lookups.md +++ b/docs/preview_compact_lookups.md @@ -118,15 +118,27 @@ the v1 view. ### Ingest safety -The new views run inside every `events_local` flush. Production uses -`async_insert=1, wait_for_async_insert=0`, so the ingester never sees a flush -error. A view that throws would fail the whole flush, and the rows would be -lost silently. +The new views run inside every `events_local` insert, so a view that throws +fails the whole insert. + +Production (checked 2026-10-10) inserts **synchronously**: +- `async_insert = 0` for the ingest user; +- about 70 sealed-segment inserts per hour; +- `system.asynchronous_insert_log` is empty. + +The `async_insert` profile settings in `ops/production/clickhouse/config.xml` +are not in effect: ClickHouse reads profiles from the users configuration, +not from a server `config.d` file. A failing view would therefore fail the +INSERT. The ingester would see the error, retry the segment and then queue it +for reindexing. Rows are not lost silently, but ingest into ClickHouse stalls +until the view is fixed or dropped. If async inserts with +`wait_for_async_insert=0` are ever enabled, a failing view would instead drop +rows silently. Precautions: - Every view expression is non-throwing: `OrZero` conversions, array indexing that returns defaults, and no fallible casts. -- An integration test pushes malformed tags through production async-insert settings and asserts that every row lands and `system.asynchronous_insert_log` shows no failures. -- Monitoring queries are in the runbook. +- Integration tests push malformed tags through both synchronous and async (`wait_for_async_insert=0`) inserts and assert that every row lands. +- The runbook monitors `system.query_views_log` (per-view status) and insert exceptions in `system.query_log`. It has an abort rule that drops the views. ## Behaviour differences from the legacy reads @@ -293,7 +305,7 @@ production build) prove correctness and mechanics only. 3. **Backfill impact:** runtime and impact on live ingest and queries at the chosen `--max-threads`/`--max-memory-gib`. 4. **Ingest overhead:** flush latency and errors after the views are created - (`system.asynchronous_insert_log`, `system.query_views_log`). + (`system.query_views_log`, insert exceptions in `system.query_log`). 5. **Latency:** `bench` with projections off, query cache off and `--drop-caches` in a quiet window. The sub-second target is met only if this run shows it. diff --git a/ops/RUNBOOK.md b/ops/RUNBOOK.md index cae344b..819fc65 100644 --- a/ops/RUNBOOK.md +++ b/ops/RUNBOOK.md @@ -406,7 +406,9 @@ just ch-query "SELECT name, count() parts, sum(rows) rows, formatReadableSize(su just ch-query "SELECT count() FROM system.parts p WHERE database = 'nostr' AND table = 'events_local' AND active AND name NOT IN (SELECT parent_name FROM system.projection_parts WHERE database = 'nostr' AND table = 'events_local' AND name = 'events_by_kind' AND active)" just ch-query "SELECT name, formatReadableSize(unreserved_space) unreserved, formatReadableSize(total_space) total FROM system.disks" just ch-query "SELECT count() FROM system.mutations WHERE database = 'nostr' AND table = 'events_local' AND is_done = 0" -just ch-query "SELECT count() FROM system.asynchronous_insert_log WHERE event_date = today()" +# Ingest inserts: expect ~70/hour, synchronous (async_insert empty or 0), no errors: +just ch-query "SELECT toStartOfHour(event_time) h, count() n, sum(written_rows) rows, any(Settings['async_insert']) async_setting, countIf(exception != '') errors FROM system.query_log WHERE event_time > now() - INTERVAL 3 HOUR AND query_kind = 'Insert' AND has(tables, 'nostr.events_local') AND type != 'QueryStart' GROUP BY h ORDER BY h" +just ch-query "SELECT view_name, status, count() FROM system.query_views_log WHERE event_time > now() - INTERVAL 1 HOUR GROUP BY view_name, status ORDER BY view_name" ``` Do not continue if `events_by_kind` is missing from any part or a mutation @@ -419,17 +421,23 @@ just ch-migrate docs/migrations/prepared/017_preview_compact_lookups.sql ``` This creates empty tables and three insert-triggered views on `events_local`. -For the next 30 minutes, watch the ingest flushes and the views: +For the next 30 minutes (about 35 ingest inserts), watch the views and the +inserts: ```bash -just ch-query "SELECT status, count(), any(exception) FROM system.asynchronous_insert_log WHERE event_time > now() - INTERVAL 30 MINUTE AND database = 'nostr' AND table = 'events_local' GROUP BY status" -just ch-query "SELECT view_name, status, count(), any(exception) FROM system.query_views_log WHERE event_time > now() - INTERVAL 30 MINUTE AND view_name LIKE 'nostr.%' GROUP BY view_name, status" +just ch-query "SELECT view_name, status, count(), any(exception) FROM system.query_views_log WHERE event_time > now() - INTERVAL 30 MINUTE AND view_name LIKE 'nostr.%' GROUP BY view_name, status ORDER BY view_name" +just ch-query "SELECT count() n, countIf(exception != '') errors, any(exception) FROM system.query_log WHERE event_time > now() - INTERVAL 30 MINUTE AND query_kind = 'Insert' AND has(tables, 'nostr.events_local') AND type != 'QueryStart'" journalctl -u pensieve-ingest --since "30 min ago" | grep -i clickhouse ``` -**Abort:** if any flush status is not `Ok`, or any view reports an -exception, run the rollback SQL immediately (step 7). A failing view drops -ingest rows silently under `wait_for_async_insert=0`. +Each new view should show `QueryFinish` with the same count as the existing +views. + +**Abort:** if any view reports an exception, or inserts into `events_local` +start failing, run the rollback SQL immediately (step 7). Production inserts +are synchronous, so a failing view fails the insert. The ingester then +retries and queues the segment for reindexing, and ClickHouse ingest stalls +until the view is dropped. ### 3. Plan and estimate disk From 7f058046a08ea58a64193dee22d6c948eb3530df Mon Sep 17 00:00:00 2001 From: Jeff Gardner <202880+erskingardner@users.noreply.github.com> Date: Sat, 10 Oct 2026 15:16:10 -0500 Subject: [PATCH 6/6] Limit addressable_latest to kinds nstr.to previews Production sizing (2026-10-10) found 646M rows across all replaceable and addressable kinds, dominated by bulk machine kinds (30382 alone is 219M), so backfilling all of them would need ~188 GiB of worst-case headroom against 256 GiB free. naddr demand was 4 previews in 7 days, all kind 30023, and the legacy naddr query never used events_by_kind. Cover only shareable content kinds (articles, video, live events, wiki, publications, listings, calendar, communities, marketplace, curation sets, git repos, music, starter packs): about 12.7M rows and 2.9M coordinates. Other naddr kinds keep the legacy query. The kind list lives in one Rust constant and a unit test requires migration 017's view to filter on exactly that list. Co-Authored-By: Claude Opus 5.5 --- .../pensieve-preview/src/lookups/compact.rs | 30 +++++++------- crates/pensieve-preview/src/lookups/target.rs | 40 +++++++++++++++++-- .../pensieve-preview/tests/compact_lookups.rs | 20 +++++++--- docs/migrations/README.md | 5 ++- .../prepared/017_preview_compact_lookups.sql | 12 +++++- docs/preview_compact_lookups.md | 23 +++++++++-- ops/RUNBOOK.md | 10 ++++- 7 files changed, 105 insertions(+), 35 deletions(-) diff --git a/crates/pensieve-preview/src/lookups/compact.rs b/crates/pensieve-preview/src/lookups/compact.rs index 270dbc8..ba9f39b 100644 --- a/crates/pensieve-preview/src/lookups/compact.rs +++ b/crates/pensieve-preview/src/lookups/compact.rs @@ -15,10 +15,13 @@ use serde::Deserialize; use crate::error::PreviewError; use crate::query::{EventRow, ProfileRow, ZapTotals, display_names_from_profiles}; -/// Whether `addressable_latest` covers a kind (replaceable 10000-19999 and -/// addressable 30000-39999). Other naddr kinds use the legacy query. +/// Whether `addressable_latest` covers a kind (see +/// [`ADDRESSABLE_KINDS`](super::target::ADDRESSABLE_KINDS)). Other naddr kinds +/// use the legacy query. pub fn addressable_kind_covered(kind: u16) -> bool { - (10_000..20_000).contains(&kind) || (30_000..40_000).contains(&kind) + super::target::ADDRESSABLE_KINDS + .binary_search(&kind) + .is_ok() } /// Fetch the latest profile (kind 0) for a pubkey from `profile_latest`. @@ -159,17 +162,14 @@ mod tests { use super::*; #[test] - fn addressable_coverage_bounds() { - assert!(!addressable_kind_covered(0)); - assert!(!addressable_kind_covered(9_999)); - assert!(addressable_kind_covered(10_000)); - assert!(addressable_kind_covered(10_002)); - assert!(addressable_kind_covered(19_999)); - assert!(!addressable_kind_covered(20_000)); - assert!(!addressable_kind_covered(29_999)); - assert!(addressable_kind_covered(30_000)); - assert!(addressable_kind_covered(30_023)); - assert!(addressable_kind_covered(39_999)); - assert!(!addressable_kind_covered(40_000)); + fn addressable_coverage_is_the_previewable_list() { + for kind in [30023, 30024, 34235, 34236, 30311, 30818, 34550, 39089] { + assert!(addressable_kind_covered(kind), "{kind}"); + } + // Bulk machine kinds, replaceable lists and non-addressable kinds fall + // back to the legacy query. + for kind in [0, 1, 10002, 30000, 30078, 30304, 30382, 30383, 40000] { + assert!(!addressable_kind_covered(kind), "{kind}"); + } } } diff --git a/crates/pensieve-preview/src/lookups/target.rs b/crates/pensieve-preview/src/lookups/target.rs index 3dc1919..67e0449 100644 --- a/crates/pensieve-preview/src/lookups/target.rs +++ b/crates/pensieve-preview/src/lookups/target.rs @@ -21,9 +21,19 @@ pub enum Target { Addressable, } -/// The kind predicate for replaceable and addressable events. -const ADDRESSABLE_KINDS: &str = - "((kind >= 10000 AND kind < 20000) OR (kind >= 30000 AND kind < 40000))"; +/// Addressable kinds covered by `addressable_latest`, sorted: shareable +/// content that nstr.to previews via naddr (articles, video, live events, +/// wiki, publications, listings, calendar, communities, marketplace, +/// curation sets, git repos, music, starter packs). Bulk machine kinds such +/// as 30382/30304/30078 and relay lists are excluded; their naddr previews use +/// the legacy query. Must match the view's `WHERE` in migration 017. +pub const ADDRESSABLE_KINDS: [u16; 20] = [ + 30004, 30005, 30017, 30018, 30023, 30024, 30040, 30041, 30311, 30402, 30403, 30617, 30818, + 31337, 31922, 31923, 34235, 34236, 34550, 39089, +]; + +/// SQL form of [`ADDRESSABLE_KINDS`]. +const ADDRESSABLE_KINDS_PREDICATE: &str = "kind IN (30004, 30005, 30017, 30018, 30023, 30024, 30040, 30041, 30311, 30402, 30403, 30617, 30818, 31337, 31922, 31923, 34235, 34236, 34550, 39089)"; /// First `d` tag value, identical to the `events_local.d_tag` MATERIALIZED /// expression. @@ -72,7 +82,7 @@ impl Target { match self { Self::Profiles => "kind = 0", Self::Zaps => "kind = 9735", - Self::Addressable => ADDRESSABLE_KINDS, + Self::Addressable => ADDRESSABLE_KINDS_PREDICATE, } } @@ -237,6 +247,28 @@ mod tests { } } + #[test] + fn addressable_kinds_match_predicate_and_migration() { + let mut sorted = ADDRESSABLE_KINDS; + sorted.sort_unstable(); + assert_eq!(sorted, ADDRESSABLE_KINDS, "keep ADDRESSABLE_KINDS sorted"); + let expected = format!( + "kind IN ({})", + ADDRESSABLE_KINDS + .iter() + .map(u16::to_string) + .collect::>() + .join(", ") + ); + assert_eq!(ADDRESSABLE_KINDS_PREDICATE, expected); + let migration = + include_str!("../../../../docs/migrations/prepared/017_preview_compact_lookups.sql"); + assert!( + migration.contains(&format!("WHERE {expected}\nGROUP BY pubkey, kind, d_tag;")), + "migration 017 addressable view must filter on exactly {expected}" + ); + } + #[test] fn names_are_distinct() { let mut names: Vec<_> = Target::ALL.iter().map(|t| t.table()).collect(); diff --git a/crates/pensieve-preview/tests/compact_lookups.rs b/crates/pensieve-preview/tests/compact_lookups.rs index d582b8e..319b780 100644 --- a/crates/pensieve-preview/tests/compact_lookups.rs +++ b/crates/pensieve-preview/tests/compact_lookups.rs @@ -622,7 +622,7 @@ async fn addressable_latest_handles_ties_and_falls_back_for_uncovered_kinds() { event(0x30, author, 5, 30023, "other", &[&["d", "other"]]), event(0x50, author, 7, 10002, "relays", &[&["r", "wss://x"]]), event(0x60, author, 9, 30023, "no-d", &[&["d"]]), - // Kind 1 is not covered by addressable_latest. + // Kinds 1 and 10002 are not covered by addressable_latest. event(0x70, author, 9, 1, "note", &[&["d", "n"]]), ]; for e in &events { @@ -645,11 +645,13 @@ async fn addressable_latest_handles_ties_and_falls_back_for_uncovered_kinds() { .unwrap() .unwrap(); assert_eq!(other.id, hex(0x30)); - let relays = compact::fetch_replaceable(c, 10002, &pk, "") - .await - .unwrap() - .unwrap(); - assert_eq!(relays.id, hex(0x50)); + // Relay lists are not a covered kind: no compact row is stored. + assert!( + compact::fetch_replaceable(c, 10002, &pk, "") + .await + .unwrap() + .is_none() + ); let no_d = compact::fetch_replaceable(c, 30023, &pk, "") .await .unwrap() @@ -670,6 +672,12 @@ async fn addressable_latest_handles_ties_and_falls_back_for_uncovered_kinds() { .unwrap() .unwrap(); assert_eq!(note.id, hex(0x70)); + let relays = lookups + .fetch_replaceable(10002, &pk, "") + .await + .unwrap() + .unwrap(); + assert_eq!(relays.id, hex(0x50)); let via_dispatch = lookups .fetch_replaceable(30023, &pk, "slug") .await diff --git a/docs/migrations/README.md b/docs/migrations/README.md index e0dc4b8..629f998 100644 --- a/docs/migrations/README.md +++ b/docs/migrations/README.md @@ -477,8 +477,9 @@ It creates three tables, each fed by an insert-triggered view on - `profile_latest`: latest kind 0 per pubkey; - `zap_receipts_by_target`: one row per (target event, zap receipt), using the migration 009 amount rules; -- `addressable_latest`: latest event id per (pubkey, kind, d_tag) for kinds - 10000–19999 and 30000–39999. +- `addressable_latest`: latest event id per (pubkey, kind, d_tag) for the + addressable kinds nstr.to previews via naddr (articles, video, live events, + wiki, listings, calendar and similar; bulk machine kinds are excluded). Reads stay correct before merges, with replays, and with out-of-order arrivals. The migration does **not** backfill or touch the projection. diff --git a/docs/migrations/prepared/017_preview_compact_lookups.sql b/docs/migrations/prepared/017_preview_compact_lookups.sql index dc75c8b..c80badb 100644 --- a/docs/migrations/prepared/017_preview_compact_lookups.sql +++ b/docs/migrations/prepared/017_preview_compact_lookups.sql @@ -124,7 +124,15 @@ WHERE kind = 9735; -- NADDR LOOKUP: latest event pointer per (pubkey, kind, d_tag) -- ============================================================================= -- --- Covers replaceable (10000-19999) and addressable (30000-39999) kinds. Stores +-- Covers addressable kinds that nstr.to previews via naddr: articles +-- (30023/30024), video (34235/34236), live events (30311), wiki (30818), +-- publications (30040/30041), listings (30402/30403), calendar (31922/31923), +-- communities (34550), marketplace (30017/30018), curation sets +-- (30004/30005), git repos (30617), music (31337) and starter packs (39089). +-- Bulk machine kinds (e.g. 30382/30304/30078, ~646M rows in production) and +-- replaceable lists are excluded; those naddr previews keep the legacy query. +-- The list must match ADDRESSABLE_KINDS in +-- crates/pensieve-preview/src/lookups/target.rs (a unit test enforces it). Stores -- only the winning event id; previews fetch the row from events_local by -- primary key. d_tag uses the same expression as the events_local MATERIALIZED -- d_tag column so lookups match the legacy `d_tag = ?` filter exactly. Same @@ -147,7 +155,7 @@ AS SELECT argMinState(id, CAST((-toInt64(created_at), id), 'Tuple(Int64, String)')) AS latest, max(indexed_at) AS last_indexed_at FROM nostr.events_local -WHERE (kind >= 10000 AND kind < 20000) OR (kind >= 30000 AND kind < 40000) +WHERE kind IN (30004, 30005, 30017, 30018, 30023, 30024, 30040, 30041, 30311, 30402, 30403, 30617, 30818, 31337, 31922, 31923, 34235, 34236, 34550, 39089) GROUP BY pubkey, kind, d_tag; -- ============================================================================= diff --git a/docs/preview_compact_lookups.md b/docs/preview_compact_lookups.md index 37d47af..4a2f878 100644 --- a/docs/preview_compact_lookups.md +++ b/docs/preview_compact_lookups.md @@ -110,11 +110,26 @@ the v1 view. ### `addressable_latest` -`AggregatingMergeTree ORDER BY (pubkey, kind, d_tag)` covering kinds -10000–19999 and 30000–39999. +`AggregatingMergeTree ORDER BY (pubkey, kind, d_tag)` covering the +addressable kinds nstr.to previews via naddr (`ADDRESSABLE_KINDS` in +`crates/pensieve-preview/src/lookups/target.rs`): +- articles (30023/30024) and video (34235/34236); +- live events (30311), wiki (30818) and publications (30040/30041); +- listings (30402/30403), calendar (31922/31923) and communities (34550); +- marketplace (30017/30018) and curation sets (30004/30005); +- git repos (30617), music (31337) and starter packs (39089). + +**Why the list is narrow.** Production on 2026-10-10 had 646M rows across all +of 10000–19999 and 30000–39999. Bulk machine kinds dominated them (30382 alone +was 219M, then 30304, 30383 and 30078), and backfilling all of it would have +needed ~188 GiB of worst-case headroom. The listed kinds are about 12.7M rows +and 2.9M coordinates. Actual naddr demand was 4 requests in 7 days, all kind +30023. The legacy naddr query never used `events_by_kind` (it reads the base +table through the pubkey bloom filter), so naddr does not block projection +removal either way. - **Winner rule:** the same `argMin` rule as profiles, but storing only the winning event id. The preview then reads the row from `events_local` in one query (`WHERE id IN (SELECT argMinMerge(latest) ...)`), which is a primary-key read. - **`d_tag`:** uses the same expression as the `events_local.d_tag` MATERIALIZED column, so lookups match the legacy `d_tag = ?` filter exactly. -- **Uncovered kinds:** naddr kinds outside the covered ranges use the legacy query. +- **Uncovered kinds:** naddr kinds outside the list use the legacy query. ### Ingest safety @@ -300,7 +315,7 @@ production build) prove correctness and mechanics only. `events_by_author` present? Confirm that every active part has `events_by_kind` before relying on it for the backfill. 2. **Sizing:** row and key counts per target from `plan`, especially - `addressable_latest` (kind 30078 app data may be large), the peak estimate + `addressable_latest` (now limited to previewable kinds), the peak estimate against `/data` unreserved space, and calibration after the first shards. 3. **Backfill impact:** runtime and impact on live ingest and queries at the chosen `--max-threads`/`--max-memory-gib`. diff --git a/ops/RUNBOOK.md b/ops/RUNBOOK.md index 819fc65..bfc7e0a 100644 --- a/ops/RUNBOOK.md +++ b/ops/RUNBOOK.md @@ -446,8 +446,14 @@ $PL plan --max-rows-per-shard 2000000 --min-free-gib 100 ``` For each target, check source rows, keys, final and peak size, and the -combined peak against `/data` unreserved space. If `addressable` is -unexpectedly large, plan and backfill `--target profiles --target zaps` first. +combined peak against `/data` unreserved space. Production measurements from +2026-10-10, for comparison: +- profiles: 53.8M rows, 18.4M keys, ~8 GiB final; +- zaps: 7.2M receipts, ~0.6 GiB; +- addressable: about 12.7M rows, 2.9M coordinates. + +Free space was 256 GiB. If a target comes out far larger than this, stop and +re-check before backfilling it. ### 4. Backfill (while `events_by_kind` exists)