pgstac-rs: Rust engine for pgstac (search, ingest, dump/export) by bitner · Pull Request #458 · stac-utils/pgstac

bitner · 2026-06-30T15:51:42Z

Major reworking of io functions and tooling for pgstac.

Add rustac based rust library with python bindings and cli for all public pgstac functionality
SQL functions and rust tooling to enable streaming data from searches which improves performance and massively reduces memory for search returns.
Remove the additional constraints that were used to enhance partition pruning. Replaced with an indexed lookup to a statistics table. This reduces lock contention and adds the ability to further prune partitions by spatial extent as well as maintaining the performance previously gained from the constraint on end_datetime.
Numerous changes to increase robustness under heavy concurrency both on the SQL side and the Rust side.

bitner · 2026-06-30T19:25:23Z

Current benchmarking run on local laptop

TTLR = Time to Last Record
TTFR = Time to First Record (really only applicable to the ndjson streaming output from rust)
911h = v0.9.11 running sql search()
911nh = v0.9.11 running sql search() with nohydrate
010sql = v0.10.0 running sql search()
010ic = v0.10.0 using rust streaming behind the scenes to build a FeatureCollection
ttfr/ttlr = v0.10.0 using rust streaming, time to first record and time to last record
010gpq = v0.10.0 streaming arrow batches to stac-geoparquet

Note: in all instances that return a full FeatureCollection, the entire set must be held in memory. Rust streaming in v0.10.0 stays very low constant memory. Rust streaming to geoparquet stays constant memory, but memory usage could still be high depending on row group size settings. Also, note that 911nh is still unhydrated, so any use of that path would still require the additional client side hydration, so it is not entirely apples to apples.

pgstac search benchmark — bench_v0911 (0.9.11) vs bench_current (0.10), 100k items/collection

All values are TTLR ms (lower = faster); rust column is NDJSON-stream TTFR/TTLR; 010gpq writes parquet.

filter	limit	911h	911nh	010sql	010ic	rust TTFR/TTLR	010gpq
broad	10	190	58	92	87	53/55	60
bbox CONUS	10	199	82	126	55	52/54	70
intersects poly	10	184	57	107	55	43/49	73
landsat 1-month	10	165	57	83	51	62/64	75
cql2 cloud<20	10	174	70	91	64	58/60	81
sort cloud asc	10	982	650	182	95	87/91	87
fields id,geom	10	62	58	57	47	44/45	48
fields -assets	10	69	74	68	53	53/55	55
broad	100	1097	103	195	122	44/59	125
bbox CONUS	100	1275	176	252	111	68/83	160
intersects poly	100	1082	148	237	129	56/91	185
landsat 1-month	100	1113	93	197	113	51/62	137
cql2 cloud<20	100	842	89	167	104	50/61	100
sort cloud asc	100	1338	658	212	134	83/97	146
fields id,geom	100	99	92	105	44	51/56	56
fields -assets	100	138	128	264	61	61/74	86
broad	1000	10768	560	1290	654	50/140	841
broad	10000	102649	5914	11841	6040	66/984	6029

PAGING — total ms to fetch K pages of L items via keyset tokens, vs one streaming pass of K*L

filter	K x L	911 pages	010 pages	010ic pages	stream K*L
broad	5 x 100	5363	939	637	102
bbox CONUS	5 x 100	5514	967	615	116
cql2 cloud<20	5 x 100	4195	843	549	95

INGEST throughput — v0.9.11 pypgstac load vs the 0.10 Rust loader (from ingest_bench.sh, 100k/coll)

collection	v0.9.11 ndjson	Rust ndjson	speedup	Rust parquet
naip 89.8k	8,394 it/s	26,758 it/s	3.2x	20,310 it/s
landsat 100k	1,695 it/s	4,231 it/s	2.5x	2,800 it/s
sentinel 100k	2,710 it/s	5,948 it/s	2.2x	4,191 it/s

… (squash of v010-export-search onto main) main already carries the v0.10 foundation (#438 rust crate, #448 fragments, #456 keyset). This adds the v010-export-search work on top: the pgstac-rs export/dump/restore engine (Rust loader, search_plan search, dump/export) with Python bindings + the pgstac CLI; flat-memory CLI search streaming via the server portal with fields projection; configurable parquet row-group size (builder, --row-group-size, PGSTAC_PARQUET_ROW_GROUP_SIZE env, Python search_to_geoparquet); SQL maintenance/comment cleanup; pypgstac updates. Foundation overlaps resolved to the developed v010io versions. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

gadomski

I didn't do a detailed review yet, just looked at some basic things. I've got a high level question about whether there's more stuff we can use from the stac crate (e.g. Search).

Move memprofile from a shipped [[bin]] to examples/ (it is a dev heap profiler, not a user tool). Remove the env-gated PGSTAC_LOAD_PROFILE / PGSTAC_PIPE_PROFILE / PGSTAC_DECODE_ONLY instrumentation from the ingest and parquet-pipeline hot paths. Fix Footprint::observe to read east/north at len/2 and len/2+1 so 6-element 3D bboxes are handled correctly (the old last-two logic placed north and max-elevation into east and north). Fix the stale stream_ndjson test call to pass the new token argument. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

PgstacPool now implements stac::api::{ItemsClient, CollectionsClient, StreamItemsClient, TransactionClient}, returning stac::api::ItemCollection via a new TryFrom<Page>, so a pgstac database is a first-class rustac backend. Each method routes through the existing engine (the keyset search in crate::search and the Rust loader for writes) - the rustac-native surface over one engine, not a second implementation. Enables stac's async feature for StreamItemsClient, whose search_stream maps the crate's flat-memory portal stream into stac::api Items (constant memory, not page-by-page). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Turn the rust-crate CI job back on (it was gated off with if: false pending v0.10). Build the three template databases the tests need (clean + ingest templates and the rich fragment template) from the committed pgstac.sql and tracked fixtures via the setup scripts, then compile-check all features and run the suite with --features cli. Testing with cli rather than --all-features avoids linking libpython at runtime - the python feature is only a compile target here and has no runtime tests. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

items_staging_triggerfunc now raises if any staged item references a collection that does not exist, instead of silently dropping it in the collections JOIN below - matching the Rust loader, which already errors on this. Validated with pgtap + basicsql + pg_dump/restore (no regression). pgstac.sql reassembled via stageversion (unreleased, no version bump); the regenerated migrations are intentionally not committed (v0.10 migrations regenerate per-run and are excluded from the gate). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

…role Groups the flat crate root into db/ read/ load/ json/ api/ python/ and finishes the rustac-native + cleanup pass. Precheck stays a library method, not yet wired into load. pgstac-rs: - reorg: flat root modules into db/ read/ load/ json/ api/ python/ (public API unchanged) - search CLI dedup: route the itemcollection command through stac::api ItemsClient::search; delete SearchSource and page_to_feature_collection; map bad search tokens to Error::InvalidToken - remove the profiling feature, memprofile example, and dhat dep (preserved on the archive/memprofile branch); drop the lib.rs dhat shim - drop the EXPERIMENTAL hedge from the precheck docs; comments describe current behavior only pgstac (SQL): - remove the unused pgstac_load up-privilege role (added this cycle, never released); the SECURITY DEFINER write functions remain the only write path through the privilege wall - add a pgtap proof-test that the wall holds: pgstac_ingest cannot directly INSERT/UPDATE/DELETE items, partition_stats, or item_fragments - regenerate pgstac.sql Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

gadomski · 2026-07-01T18:18:40Z

+#[derive(ValueEnum, Clone, Copy, Debug)]
+enum CompressionArg {
+    Zstd,
+    Snappy,
+    Uncompressed,
+}


This has an enum in rustac, can we use that?

gadomski · 2026-07-01T18:19:08Z

+#[derive(ValueEnum, Clone, Copy, Debug, PartialEq, Eq)]
+enum SearchFormat {
+    /// Newline-delimited JSON, one item per line (streams all pages).
+    Ndjson,
+    /// A single STAC FeatureCollection page (SQL-faithful next/prev + context).
+    Itemcollection,
+    /// stac-geoparquet of the (bounded) result.
+    Geoparquet,
+}


Same as below, can we re-use the enum from rustac?

…gnore Opt-in --skip-unchanged toggle on `load` routes each batch through the per-partition precheck before loading: on re-ingest / sync it skips items already present-and-current and loads only new + changed. Benchmarks (NAIP 89.8k, release): re-ingest identical ~3.8-5.1x faster, 10%-changed ~3.3x; ~18% slower on an all-new load, so it stays opt-in. - policy-aware precheck: ignore/error skip an existing id by id alone (no content hash, and skip changed-existing too, matching ignore semantics); upsert/delsert keep the content-hash compare - error + --skip-unchanged is rejected (error must fail on an existing id, not skip it) - the survivor-count preflight bump makes partition_stats.n tighter on a precheck load (ignore exact, upsert new+changed) with no extra query and no stats-on-flush - concurrency test covering the id-only ignore path Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

gadomski

This was a API-level review, meaning that I looked through the publicly exposed functions (via the documentation). I have not looked through the function bodies, except when I found myself digging deeper.

There's a lot of free functions that feel like they could/should be methods provided by the Pgstac trait or the Client struct, since they take a client as their first argument?

Also, Claude flagged up a couple of issues:

pgstac/src/pypgstac/src/pypgstac/load.py

Line 480 in 4400bfd

cur.execute("SELECT update_partition_stats_q(%s);", (partition.name,))

is broken as update_partition_stats_q doesn't exist anymore
Is it intentional that partition_stats isn't backfilled during a migration?
Is indexes_pending used anywhere to build the queryable indexes?

gadomski · 2026-07-02T13:02:36Z

+#[derive(Debug, Clone)]
+pub struct WkbGeometry(pub Vec<u8>);


Duplicative on the geometry from the geom module?

gadomski · 2026-07-02T13:08:36Z

+#[derive(Debug, Clone)]
+pub struct ConnectConfig {
+    /// A full connection string (libpq keyword/value or `postgresql://` URL). When set it is the base
+    /// configuration; the individual fields below still override or augment it.
+    pub dsn: Option<String>,
+    /// Database host (`PGHOST`).
+    pub host: Option<String>,
+    /// Database port (`PGPORT`).
+    pub port: Option<u16>,
+    /// Database name (`PGDATABASE`).
+    pub dbname: Option<String>,
+    /// User (`PGUSER`).
+    pub user: Option<String>,
+    /// Password (`PGPASSWORD`).
+    pub password: Option<String>,
+    /// Extra libpq `options` startup string (`PGOPTIONS`); the search path is merged into it.
+    pub options: Option<String>,
+    /// Application name (`PGAPPNAME`); defaults to [`DEFAULT_APPLICATION_NAME`].
+    pub application_name: Option<String>,
+    /// Connection timeout in seconds (`PGCONNECT_TIMEOUT`).
+    pub connect_timeout: Option<u64>,
+    /// SSL mode (`PGSSLMODE`): `disable`, `allow`, `prefer` (the default), `require`, `verify-ca`, or
+    /// `verify-full`.
+    pub sslmode: Option<String>,
+    /// Path to a trusted CA certificate (`PGSSLROOTCERT`); consumed by the TLS connector.
+    pub sslrootcert: Option<String>,
+    /// Path to a client certificate (`PGSSLCERT`); consumed by the TLS connector.
+    pub sslcert: Option<String>,
+    /// Path to a client private key (`PGSSLKEY`); consumed by the TLS connector.
+    pub sslkey: Option<String>,
+    /// The search path applied at connection startup. Defaults to [`DEFAULT_SEARCH_PATH`].
+    pub search_path: String,
+}


This feels really duplicative on stuff that already exists for postgres, do we really need it?

gadomski · 2026-07-02T13:09:20Z

+/// A page of search results.
+#[derive(Debug, Deserialize, Serialize)]
+pub struct Page {
+    /// These are the out features, usually STAC items, but maybe not legal STAC
+    /// items if fields are excluded.
+    pub features: Vec<Item>,
+
+    /// The next id.
+    #[serde(skip_serializing_if = "Option::is_none")]
+    pub next: Option<String>,
+
+    /// The previous id.
+    #[serde(skip_serializing_if = "Option::is_none")]
+    pub prev: Option<String>,
+
+    /// The search context.
+    ///
+    /// This was removed in pgstac v0.9
+    #[serde(skip_serializing_if = "Option::is_none")]
+    pub context: Option<Context>,
+
+    /// The number of values returned.
+    ///
+    /// Added in pgstac v0.9
+    #[serde(rename = "numberReturned", skip_serializing_if = "Option::is_none")]
+    pub number_returned: Option<usize>,
+
+    /// Links
+    ///
+    /// Added in pgstac v0.9
+    #[serde(default, skip_serializing_if = "Vec::is_empty")]
+    pub links: Vec<Link>,
+
+    /// Additional fields.
+    #[serde(flatten)]
+    pub additional_fields: Map<String, Value>,
+}


Is this duplicative on SearchPage?

The precheck path takes `&PgstacPool`, so it only compiles with the `pool` feature, but CI only ran `cargo check --all-features` — so the no-`pool` build was broken (E0425: cannot find `PgstacPool`). Gate the precheck items and their helpers behind `#[cfg(feature = "pool")]`, gate the two export-only integration tests, and replace the single all-features check with a feature-matrix check (no-default / pool / export / cli / all-features) so the break cannot recur. Per the #458 review. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

… in CI cargo doc emitted 5 broken-intra-doc-link warnings — links to private / non-Rust items (precheck_one_partition, hydrate_fragment_core, PgConn, stac_daterange) become plain code, and GenericClient links to its real tokio_postgres path. Also refresh the crate features list (pool/export/cli/python; the old `tls` entry was stale) and the export module wording. Add a RUSTDOCFLAGS=-D warnings `cargo doc` CI step so doc warnings cannot regress. Per the #458 review. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

The SQL comments called jsonb_canonical/jsonb_hash "RFC 8785-aligned" and pointed external implementers at the `rfc8785` crate / a generic RFC 8785 canonicalizer — but the actual form is a custom canonicalization (byte-order keys via COLLATE "C" + `float8::text` numbers) that is NOT RFC 8785 and would hash differently. Correct the comments so nobody reimplements the hash against the wrong spec. The Rust side already documented this correctly, and parity is CI-gated by tests/canonical_parity.rs. Comment-only; pgstac.sql regenerated. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

…med constants - CLI --policy uses `ConflictPolicy` directly (cli-gated `ValueEnum` derive), dropping the duplicate `LoadPolicy` enum + its `From` impl; `delsert` is now also a CLI policy. - The Int queryable dehydrate uses `i32::try_from` (erroring) instead of `as i32`, so an out-of-range integer property fails loudly instead of silently truncating. - `DEFAULT_BATCH_SIZE` / `MAX_INGEST_PARALLELISM` constants replace the inline 5000 / 8. Per the #458 review. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

…async sweep New partitions are created index-light (indexes_pending=true) for fast ingest, but nothing consumed the flag. Wire it end-to-end: - the queryables trigger now flags every partition (SECURITY DEFINER, so it works for pgstac_ingest past the partition_stats write wall) instead of rebuilding all partitions synchronously in the trigger; - a new build_pending_indexes(_limit) maintenance sweep builds the missing per-partition queryable indexes and clears the flag, mirroring tighten_dirty_partition_stats (schedule via pg_cron / the maintenance CLI). Per the #458 review (indexes_pending was set but never consumed). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

CREATE DATABASE ... TEMPLATE copies the schema but not the per-database settings (pg_db_role_setting) of the template, so each cloned per-test DB lost the search_path = pgstac, public the template sets — and every Pgstac-trait test that uses an unqualified pgstac reference failed in CI (pgstac_version etc. passed because they are qualified). Re-apply the search_path on the clone. Full cargo test --features cli is green, matrix + docs clean. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Having both the `Pgstac` trait (a bare-connection trait returning `Page`/`JsonValue`) and `Client`/`PgstacPool` (the `stac::api` traits returning stac types) was two public ways to do the same thing. The trait has no external consumers except rustac, so remove it entirely: - low-level SQL-call helpers move to `db::call` (pub(crate) free fns); the 3 unused ones (pgstac_vec/opt/value) are dropped - version/settings/collection-maintenance/delete_item become inherent `Client` methods; search/item/collections/collection/add_collection stay on the `stac::api` impls - `ItemsClient::search` folds in the malformed-token to InvalidToken mapping and uses the existing `TryFrom<Page>` instead of a hand-rolled ItemCollection rebuild (also carrying number_matched/number_returned/links the old code dropped) - the pool search/collections impls delegate to `Client` (one impl, shared cache) - `Pgstac` and `Page` leave the public API; the crate docs/examples lead with `Client`; the ~80 test sites move onto the Client surface - net -140 lines Engines (ingest/search/stream/hydrate) are untouched; a pre/post A/B shows no perf regression and streaming is unchanged. Follow-up (separate rustac PR): migrate rustac crates/server/src/backend/pgstac.rs from the `Pgstac` trait to the `stac::api` traits on `PgstacPool`/`Client`. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

`ConnectConfig::from_env` reads `PGDATABASE` (and `PGHOST`/`PGPORT`/...) into the config, and `to_pg_config` applied those over a later-set `--dsn` — so `pgstac load --dsn <db>` connected to `PGDATABASE` instead of the database the DSN named. In CI (PostgreSQL 18, `PGDATABASE=postgis`) the CLI load tests failed with `function upsert_collection(jsonb, text) does not exist`, because the CLI connected to `postgis` (no pgstac) rather than the cloned test database. Locally it was masked by an empty `PGDATABASE` and a pgstac-first cluster `search_path`. Add `ConnectConfig::with_dsn`, which makes an explicit DSN the authoritative connection target (clearing host/port/dbname/user/password from the environment while keeping TLS/options/search_path), and route the CLI `--dsn` flag through it. Verified on a local PG18 container with `PGDATABASE=postgis`. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

gadomski

Some comments from my previous PR review need to be addressed, e.g. around enum re-use and using some stac structures
Got a local error running the test canonical_and_hash_match_sql: function pgstac.jsonb_canonical_hash(jsonb) does not exist

gadomski · 2026-07-08T17:27:14Z

+    /// Postgres connection string. Defaults to $PGSTAC_DSN or a local dev DSN.
+    #[arg(long, env = "PGSTAC_DSN")]
+    dsn: Option<String>,
+


You can use global = true in arg to share dsn among all the argument structures, e.g. https://github.com/stac-utils/rustac/blob/main/crates/cli/src/lib.rs#L43C1-L57C34

gadomski · 2026-07-08T17:28:37Z

+    /// Item id. Given, deletes that item; omitted, deletes the whole collection (and its items).
+    #[arg(long)]
+    item: Option<String>,
+}


Since delete is a dangerous operation, should we add a confirmation prompt by default that could be disabled with a boolean flag?

gadomski · 2026-07-08T17:32:57Z

+/// value (unlimited) is treated as no cgroup limit.
+fn cgroup_memory_limit() -> Option<u64> {
+    // cgroup v2
+    if let Ok(s) = std::fs::read_to_string("/sys/fs/cgroup/memory.max") {


This feels pretty gnarly — does it work on non-linux systems?

gadomski · 2026-07-08T17:45:20Z

+    fn put(&self, rel_path: &str, bytes: &[u8]) -> Result<FileWritten>;
+
+    /// Writes a finished large file by streaming it from a local path.
+    fn put_file(&self, rel_path: &str, src: &Path) -> Result<FileWritten>;


Suggested change

fn put_file(&self, rel_path: &str, src: &Path) -> Result<FileWritten>;

fn put_from_path(&self, rel_path: &str, src: &Path) -> Result<FileWritten>;

gadomski · 2026-07-08T17:47:15Z

+// object_store sink (local dir + S3/GCS/Azure)
+// ---------------------------------------------------------------------------
+
+#[cfg(feature = "cli")]


Since this doesn't have to be just for the CLI, to me this should be behind an object_store (or store) feature, and then the cli feature can depend on the object_store feature.

gadomski · 2026-07-08T17:48:03Z

+    pub struct ObjectStoreSink {
+        store: Arc<dyn ObjectStore>,
+        base_prefix: ObjPath,
+        runtime: tokio::runtime::Handle,


Why does this own a runtime? Can we just make the methods async on the trait?

bitner requested a review from gadomski June 30, 2026 16:08

bitner force-pushed the v010io branch from a5783db to 00491e4 Compare June 30, 2026 18:54

bitner force-pushed the v010io branch from 00491e4 to f3830b5 Compare June 30, 2026 19:29

gadomski reviewed Jun 30, 2026

View reviewed changes

Comment thread src/pgstac-rs/Cargo.toml Outdated

Comment thread src/pgstac-rs/src/lib.rs Outdated

Comment thread docs/src/pgstac-rs.md

Comment thread src/pgstac-rs/src/export/mod.rs Outdated

Comment thread src/pgstac-rs/src/read/search.rs

bitner and others added 5 commits June 30, 2026 15:57

gadomski reviewed Jul 1, 2026

View reviewed changes

gadomski requested changes Jul 2, 2026

View reviewed changes

bitner and others added 8 commits July 2, 2026 10:45

bitner marked this pull request as ready for review July 6, 2026 18:00

bitner requested a review from gadomski July 6, 2026 18:00

bitner added 3 commits July 6, 2026 13:51

address issues from reviews

be02b89

fix issue in item hash parity between postgres/rust

1fc2fbc

fix tests

1723348

gadomski requested changes Jul 8, 2026

View reviewed changes

	fn put_file(&self, rel_path: &str, src: &Path) -> Result<FileWritten>;
	fn put_from_path(&self, rel_path: &str, src: &Path) -> Result<FileWritten>;

Uh oh!

Conversation

bitner commented Jun 30, 2026

Major reworking of io functions and tooling for pgstac.

Uh oh!

bitner commented Jun 30, 2026

Current benchmarking run on local laptop

pgstac search benchmark — bench_v0911 (0.9.11) vs bench_current (0.10), 100k items/collection

PAGING — total ms to fetch K pages of L items via keyset tokens, vs one streaming pass of K*L

INGEST throughput — v0.9.11 pypgstac load vs the 0.10 Rust loader (from ingest_bench.sh, 100k/coll)

Uh oh!

gadomski left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

gadomski left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gadomski left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

gadomski left a comment •

edited

Loading