Skip to content

Fix: search limit for items with same ids, different collections#461

Draft
jonhealy1 wants to merge 3 commits into
stac-utils:mainfrom
jonhealy1:fix-search-limit-same-id
Draft

Fix: search limit for items with same ids, different collections#461
jonhealy1 wants to merge 3 commits into
stac-utils:mainfrom
jonhealy1:fix-search-limit-same-id

Conversation

@jonhealy1

@jonhealy1 jonhealy1 commented Jul 4, 2026

Copy link
Copy Markdown

Technical Context

Description

Summary of Changes

This PR addresses a critical keyset pagination failure that occurs when two separate collections contain an item with the exact same id and identical sorting metadata (e.g., matching datetime timestamps).

The core fix is implemented entirely within keyset_sortkeys(_search jsonb):

  1. Tuple Expansion: Upgraded the internal sorting sequence from a doublet (datetime, id) to a globally unique triplet: (datetime, collection, id).
  2. Planner Optimization: Positioned the collection element before the item id. Because PgSTAC tables are physically partitioned by collection identifier, this arrangement allows the PostgreSQL query planner to perform partition pruning, preserving high-throughput performance over millions of rows.
  3. Registry Defensiveness: Added an explicit CASE handler inside the dynamic expression builder to map field = 'collection' safely to table column targets (i.collection), preventing it from being dropped as an unregistered NULL expression.

Rationale

In the STAC specification, an item id is only guaranteed to be unique within its parent collection catalog. When executing multi-collection searches, paginating past matching items previously forced a strict tuple inequality check in PostgreSQL:

(datetime, id) < (token_datetime, token_id)

When evaluating duplicate IDs on Page 2, this statement evaluates to FALSE (e.g., (X, Y) < (X, Y)). The database mistakenly concludes that no remaining records exist, collapsing the response payload to 0 features and corrupting the subsequent prev token link calculations. Introducing the collection partition key ensures stable, deterministic row differentiation.

Area for Reviewers to Focus On

  • src/pgstac/sql/002d_keyset.sql: Verify that the hardcoded string fallback for core tracking identifiers (id, collection) successfully prevents registry lookup misses.
  • src/pgstac/tests/pgtap/004_search.sql: Review the programmatic pgTAP integration test. It leverages jsonb_path_query_first to pull and feed the base64 token dynamically between pages to evaluate forward/reverse stability against a seeded duplicate item name without relying on brittle, hardcoded strings.

Checklist

  • Linting: Code is formatted and linted
  • Tests: Tests pass. I have included new tests for these changes where applicable.
  • Edge Cases: I have manually verified "unhappy paths" and edge cases beyond the basic success criteria (e.g., database connection timeouts, malformed input, strict mapping rejections).
  • Documentation: I have updated README.md to reflect any new environment variables, configuration changes, or breaking updates.
  • Accountability: I can explain the implementation logic for every line of code submitted.

AI tool usage

  • AI (Copilot or something similar) supported my development of this PR. See our policy about AI tool use. Use of AI tools must be indicated.

Policy: We require a "human-in-the-loop." You are the author and are fully accountable for all submitted code. Please ensure all tool-generated content is thoroughly reviewed before submission to ensure it is not an "extractive contribution" that squanders maintainer time.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

/search with ?limit fails with same item ID in different collections

1 participant