pgstac-rs: Rust engine for pgstac (search, ingest, dump/export)

.github/dependabot.yml

@@ -25,33 +25,16 @@ updates:
           - "*"
 
   - package-ecosystem: "pip"
-    directory: "/src/pypgstac"
+    directory: "/src/pgstac-migrate"
     schedule:
       interval: "weekly"
       day: "monday"
     open-pull-requests-limit: 5
     groups:
-      python-dev-tooling:
+      python-all:
         applies-to: version-updates
         patterns:
-          - "ruff"
-          - "ty"
-          - "pre-commit"
-          - "types-*"
-      python-runtime:
-        applies-to: version-updates
-        patterns:
-          - "cachetools"
-          - "fire"
-          - "hydraters"
-          - "orjson"
-          - "plpygis"
-          - "pydantic"
-          - "python-dateutil"
-          - "smart-open"
-          - "tenacity"
-          - "version-parser"
-          - "psycopg*"
+          - "*"
 
   - package-ecosystem: "cargo"
     directory: "/rust"

.github/instructions/scripts.instructions.md

@@ -14,5 +14,5 @@ See CLAUDE.md "Development Workflow" for usage. All scripts require the Docker c
 - `stageversion` regenerates `*unreleased*` migrations each run; if you hand-edit incremental SQL, rebuild the baked artifact with `uv run --directory src/pgstac-migrate pgstac-migrate build-artifact` and avoid rerunning `stageversion` unless you intend to overwrite edits
 - `scripts/container-scripts/stageversion` and `scripts/container-scripts/makemigration` now shell through `pgpkg` inside the container rather than assembling/diffing SQL directly
 - Set `PGPKG_LOCAL_REPO_DIR` on the host when you need to force a local pgpkg checkout for `stageversion`, `makemigration`, or related container-script testing
-- Tagged releases run `.github/workflows/release.yml`, which publishes both `pypgstac` and `pgstac-migrate` to PyPI via the GitHub `pypi` environment; PyPI trusted publishers must exist for both projects
+- Tagged releases run `.github/workflows/release.yml`, which publishes `pgstac-migrate` and `pypgstac-rs` to PyPI via the GitHub `pypi` environment (and the Rust crate to crates.io); PyPI trusted publishers must exist for both projects
 - DO NOT run `stageversion` without understanding its side effects

.github/workflows/continuous-integration.yml

@@ -34,7 +34,7 @@ jobs:
           filters: |
             pgstac:
               - 'docker/pgstac/**'
-            pypgstac:
+            pyrust:
               - 'docker/pypgstac/**'
       - id: check
         run: |
@@ -50,7 +50,7 @@ jobs:
           echo "buildpg=$buildpg" >>$GITHUB_OUTPUT;
           buildpy=false;
           pyref=$ref;
-          [[ "${{ steps.filter.outputs.pypgstac }}" == "true" ]] && buildpy=true || pyref=main;
+          [[ "${{ steps.filter.outputs.pyrust }}" == "true" ]] && buildpy=true || pyref=main;
           if [[ "${{ github.event_name }}" == "schedule" || "${{ github.event_name }}" == "workflow_dispatch" ]]; then
             buildpy=true;
             pyref=main;
@@ -170,60 +170,51 @@ jobs:
             exit 1
           fi
       - uses: astral-sh/setup-uv@08807647e7069bb48b6ef5acd8ec9567f424441b # v8.1.0
-      - name: Install pypgstac
-        working-directory: /__w/pgstac/pgstac/src/pypgstac
+      - name: Build the pypgstac_rs wheel and install pgstac-migrate
+        working-directory: /__w/pgstac/pgstac
         run: |
           export UV_CACHE_DIR=/tmp/.uv-cache
           export XDG_CACHE_HOME=/tmp/.cache
           mkdir -p "$UV_CACHE_DIR" "$XDG_CACHE_HOME"
           uv --cache-dir "$UV_CACHE_DIR" venv /tmp/ci-venv
-          uv --cache-dir "$UV_CACHE_DIR" pip install --python /tmp/ci-venv/bin/python ${{ matrix.flags }} .[dev,test,psycopg]
+          # Build the Rust-backed pypgstac_rs extension wheel (the read API used by stac-fastapi-pgstac).
+          uv --cache-dir "$UV_CACHE_DIR" pip install --python /tmp/ci-venv/bin/python maturin
+          /tmp/ci-venv/bin/maturin build --release -m src/pgstac-rs/Cargo.toml --out /tmp/pypgstac-rs-wheels
+          uv --cache-dir "$UV_CACHE_DIR" pip install --python /tmp/ci-venv/bin/python /tmp/pypgstac-rs-wheels/*.whl
+          # Install pgstac-migrate (the migration CLI) plus its dev/test tooling for the container test runner.
+          uv --cache-dir "$UV_CACHE_DIR" pip install --python /tmp/ci-venv/bin/python ${{ matrix.flags }} ./src/pgstac-migrate
           echo "/tmp/ci-venv/bin" >> "$GITHUB_PATH"
       - name: Run tests
         working-directory: /__w/pgstac/pgstac
         run: scripts/container-scripts/test
+  # Compile/lint only — NOT tests. The Rust *tests* run via scripts/container-scripts/test (the single
+  # entrypoint, same as local) in the `test` matrix job, against every Postgres version. This job just
+  # feature-matrix compile-checks + doc-lints once (no database needed).
   rust-crate:
-    name: Test rust crate
-    if: ${{ false }}  # FIXME: turn back on before v0.10 release
+    name: Rust compile & docs
     runs-on: ubuntu-latest
     needs:
       - changes
-      - buildpg
-    services:
-      postgres:
-        env:
-          POSTGRES_USER: username
-          POSTGRES_PASSWORD: password
-          POSTGRES_DB: postgis
-        image: ${{ needs.changes.outputs.pgtagprefix }}-pg18
-        ports:
-          - 5439:5432
-        options: >-
-          --health-cmd pg_isready
-          --health-interval 10s
-          --health-timeout 5s
-          --health-retries 5
-    env:
-      PGHOST: localhost
-      PGPORT: 5439
-      PGUSER: username
-      PGPASSWORD: password
-      PGDATABASE: postgis
     steps:
       - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
         with:
           persist-credentials: false
       - uses: Swatinem/rust-cache@e18b497796c12c097a38f9edb9d0641fb99eee32 # v2
       - uses: astral-sh/setup-uv@08807647e7069bb48b6ef5acd8ec9567f424441b # v8.1.0
-      - name: Install pypgstac
-        working-directory: src/pypgstac
+      - name: Compile-check the feature matrix
         run: |
-          uv venv /tmp/ci-venv
-          uv pip install --python /tmp/ci-venv/bin/python .[psycopg]
-          echo "/tmp/ci-venv/bin" >> "$GITHUB_PATH"
-      - name: Migrate pgstac
-        run: pypgstac migrate
-      - name: Set search_path
-        run: psql -c "ALTER ROLE username SET search_path TO pgstac, public;"
-      - name: Test
-        run: cargo test -p pgstac --all-features --manifest-path src/pgstac-rs/Cargo.toml
+          set -euo pipefail
+          for feat in \
+            "--no-default-features" \
+            "--no-default-features --features pool" \
+            "--no-default-features --features export" \
+            "--features cli" \
+            "--all-features"; do
+            echo "::group::cargo check $feat"
+            cargo check $feat --all-targets --manifest-path src/pgstac-rs/Cargo.toml
+            echo "::endgroup::"
+          done
+      - name: Check docs (deny warnings)
+        env:
+          RUSTDOCFLAGS: -D warnings
+        run: cargo doc --all-features --no-deps --manifest-path src/pgstac-rs/Cargo.toml

.github/workflows/release.yml

@@ -131,101 +131,27 @@ jobs:
           cache-from: type=gha
           cache-to: type=gha, mode=max
 
-  buildpypgstac:
-    name: Build and push base pyrust
-    runs-on: ubuntu-latest
-    steps:
-      - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
-      - uses: docker/setup-qemu-action@ce360397dd3f832beb865e1373c09c0e9f86d70a # v4.0.0
-        # Temporary workaround for ARM builds
-        # https://github.com/docker/setup-qemu-action/issues/198
-        with:
-          image: tonistiigi/binfmt:qemu-v7.0.0-28
-      - uses: docker/setup-buildx-action@d7f5e7f509e45cec5c76c4d5afdd7de93d0b3df5 # v4.1.0
-      - name: Log in to the Container registry
-        uses: docker/login-action@650006c6eb7dba73a995cc03b0b2d7f5ca915bee # v4.2.0
-        with:
-          registry: ${{ env.REGISTRY }}
-          username: ${{ github.actor }}
-          password: ${{ secrets.GITHUB_TOKEN }}
-
-      - name: Extract metadata (tags, labels) for Docker
-        id: meta
-        uses: docker/metadata-action@80c7e94dd9b9319bd5eb7a0e0fe9291e23a2a2e9 # v6.1.0
-        with:
-          images: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}-pypgstac
-      - name: Build and Push Base Postgres
-        uses: docker/build-push-action@f9f3042f7e2789586610d6e8b85c8f03e5195baf # v7.2.0
-        with:
-          platforms: linux/amd64,linux/arm64
-          context: .
-          target: pypgstac
-          file: docker/pypgstac/Dockerfile
-          tags: ${{ steps.meta.outputs.tags }}
-          labels: ${{ steps.meta.outputs.labels }}
-          push: true
-          cache-from: type=gha
-          cache-to: type=gha, mode=max
-
-  buildpypgstacruntime:
-    name: Build and push pypgstac runtime image
-    runs-on: ubuntu-latest
-    steps:
-      - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
-      - uses: docker/setup-qemu-action@ce360397dd3f832beb865e1373c09c0e9f86d70a # v4.0.0
-        with:
-          image: tonistiigi/binfmt:qemu-v7.0.0-28
-      - uses: docker/setup-buildx-action@d7f5e7f509e45cec5c76c4d5afdd7de93d0b3df5 # v4.1.0
-      - name: Log in to the Container registry
-        uses: docker/login-action@650006c6eb7dba73a995cc03b0b2d7f5ca915bee # v4.2.0
-        with:
-          registry: ${{ env.REGISTRY }}
-          username: ${{ github.actor }}
-          password: ${{ secrets.GITHUB_TOKEN }}
-
-      - name: Extract metadata (tags, labels) for Docker
-        id: meta
-        uses: docker/metadata-action@80c7e94dd9b9319bd5eb7a0e0fe9291e23a2a2e9 # v6.1.0
-        with:
-          images: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}-pypgstac-runtime
-      - name: Build and Push pypgstac runtime
-        uses: docker/build-push-action@f9f3042f7e2789586610d6e8b85c8f03e5195baf # v7.2.0
-        with:
-          platforms: linux/amd64,linux/arm64
-          context: .
-          target: pypgstac-runtime
-          file: docker/pypgstac/Dockerfile
-          tags: ${{ steps.meta.outputs.tags }}
-          labels: ${{ steps.meta.outputs.labels }}
-          push: true
-          cache-from: type=gha
-          cache-to: type=gha, mode=max
-
-  releasetopypi:
-    name: Release
+  releasepypgstacrstopypi:
+    name: Release pypgstac-rs to PyPI
     runs-on: ubuntu-latest
     permissions:
       id-token: write
     environment:
       name: pypi
-      url: https://pypi.org/p/pypgstac
+      url: https://pypi.org/p/pypgstac-rs
     steps:
       - name: Checkout
         uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
-      - name: Setup Python
-        uses: actions/setup-python@a309ff8b426b58ec0e2a45f0f869d46889d02405 # v6.2.0
-        with:
-          python-version: "3.x"
-      - name: Install build
-        working-directory: src/pypgstac
-        run: pip install build
-      - name: Build
-        working-directory: src/pypgstac
-        run: python -m build
+      - name: Setup uv
+        uses: astral-sh/setup-uv@08807647e7069bb48b6ef5acd8ec9567f424441b # v8.1.0
+      # Publishes the portable sdist so `pip install pypgstac-rs` builds the extension from source.
+      # Prebuilt manylinux/macOS/Windows wheels can be added later via a PyO3/maturin-action matrix.
+      - name: Build sdist
+        run: uvx --from maturin maturin sdist -m src/pgstac-rs/Cargo.toml --out dist
       - name: Publish to PyPI
         uses: pypa/gh-action-pypi-publish@cef221092ed1bacb1cc03d23a2d87d1d172e277b # release/v1
         with:
-          packages-dir: src/pypgstac/dist
+          packages-dir: dist
 
   releasepgstacmigratetopypi:
     name: Release pgstac-migrate to PyPI

.pre-commit-config.yaml

@@ -44,16 +44,16 @@ repos:
         pass_filenames: false
         verbose: true
         fail_fast: true
-        files: ^(src/pypgstac/(src/pypgstac|tests)/.*\.py|src/pypgstac/pyproject\.toml)$
-    -   id: pypgstac
-        name: pypgstac
+        files: ^(src/pgstac-migrate/(src/pgstac_migrate|tests)/.*\.py|src/pgstac-migrate/pyproject\.toml)$
+    -   id: pymigrate
+        name: pymigrate
         entry: scripts/test
-        args: [--pypgstac]
+        args: [--pymigrate]
         language: script
         pass_filenames: false
         verbose: true
         fail_fast: true
-        files: pypgstac\/.*\.py$
+        files: pgstac-migrate\/.*\.py$
     -   id: migrations
         name: migrations
         entry: scripts/test

AGENTS.md

@@ -25,12 +25,12 @@ Migration specialist for PgSTAC. See CLAUDE.md "Migration Process" for full work
 1. Edit SQL in `src/pgstac/sql/*.sql`
 2. `src/pgstac/pyproject.toml` is the `pgpkg` project config for the SQL + migrations tree
 3. `uv run --directory src/pgstac-migrate pgstac-migrate info|versions|plan` inspects the baked migration artifact during wrapper work
-4. `uv run --directory src/pypgstac pypgstac migrate -- --help` remains a backwards-compatible wrapper over `pgstac-migrate`; put new runtime migration behavior in `src/pgstac-migrate/`, not `src/pypgstac/`
+4. `uv run --directory src/pgstac-migrate pgstac-migrate migrate --help` drives migrations directly; put new runtime migration behavior in `src/pgstac-migrate/`
 5. `scripts/stageversion VERSION` regenerates canonical `pgstac--VERSION.sql` plus incremental `pgstac--FROM--TO.sql`; set `PGPKG_LOCAL_REPO_DIR` when `stageversion` or `makemigration` should run against a local pgpkg checkout. The Docker-backed flow mounts that override at `/pgpkg` and exports `PGPKG_REPO_DIR` to the container scripts.
 6. Review the generated incremental migration (watch for DROPs, unsafe ALTERs, missing `CREATE OR REPLACE`)
 7. If you hand-edit the incremental migration, rebuild the baked artifact: `uv run --directory src/pgstac-migrate pgstac-migrate build-artifact`
 8. Run `scripts/test --migrations` (or full `scripts/test` gate)
-9. Tagged releases publish both `pypgstac` and `pgstac-migrate` to PyPI from `.github/workflows/release.yml`; keep the PyPI trusted publisher registration aligned with the `pypi` environment and workflow path
+9. Tagged releases publish `pgstac-migrate` and `pypgstac-rs` to PyPI (and the `pgstac` crate to crates.io) from `.github/workflows/release.yml`; keep the PyPI trusted publisher registration aligned with the `pypi` environment and workflow path
 
 ### Review Checklist
 
@@ -43,14 +43,11 @@ Migration specialist for PgSTAC. See CLAUDE.md "Migration Process" for full work
 
 ## loader-developer
 
-Specialist in pypgstac bulk loading (`src/pypgstac/src/pypgstac/load.py`). See CLAUDE.md "pypgstac Loader Internals" for full details.
+Specialist in the Rust bulk loader (`src/pgstac-rs/src/load/pool_ingest.rs`, plus `src/pgstac-rs/src/load/dehydrate.rs` and `fragment.rs`). See CLAUDE.md for partition architecture.
 
 ### Critical Patterns
 
-- **Materialize generators**: `list(g)` before `load_partition()` — generators can't survive tenacity retries
-- **Live view only**: Query `partition_sys_meta` (VIEW), never `partitions` (stale MATERIALIZED VIEW)
-- **Retry safety**: `item.pop("partition", None)` with `None` default; `before_sleep` sets `partition.requires_update = True` on `CheckViolation`
-- **Retry scope**: `CheckViolation`, `DeadlockDetected`, `SerializationFailure`, `LockNotAvailable`, `ObjectInUse`
-- **Load modes**: `insert`, `ignore`/`insert_ignore`, `upsert`, `delsert`
+- **Conflict policies**: `upsert` (default), `ignore`, `error`, plus loader-level delsert
+- **Live view only**: partition metadata comes from the `partition_sys_meta` live VIEW, never the stale `partitions` MATERIALIZED VIEW
 - **Sample data load**: `scripts/loadsampledata`
-- Test: `scripts/runinpypgstac --build test --pypgstac`
+- Tests: `src/pgstac-rs/tests/` (`pool_ingest.rs`, `ingest_load.rs`, `cli.rs`); run with `scripts/test --rust`