Skip to content

Persistence: DynamoDB (atomic) listing and hasChildren - #6016

Open
ZephyrYWZhou wants to merge 5 commits into
apache:mainfrom
ZephyrYWZhou:dynamodb-atomic-listing
Open

ZephyrYWZhou wants to merge 5 commits into
apache:mainfrom
ZephyrYWZhou:dynamodb-atomic-listing

Conversation

@ZephyrYWZhou

@ZephyrYWZhou ZephyrYWZhou commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Next step of the DynamoDB (atomic) backend: listing. This implements listEntities, listFullEntities and hasChildren on DynamoDbBasePersistence, reading the strongly consistent child records on the base table.

Builds on the child records added in #6118, now merged. It replaces the earlier gsi1-based approach from the review discussion: a GSI is eventually consistent, so listing it could make a wrong correctness decision (drop-namespace emptiness, location-overlap) even for sequential requests. Child records live on the base table and are maintained in the entity's own transaction, so they can be read with ConsistentRead=true.

Changes

  • listEntities / listFullEntities: query the child records for one parent, type and subtype (PK = realmId#catalogId#parentId#typeCode#subTypeCode), in id order, with ConsistentRead=true. The child records hold only ids, so each page's ids are then read into entities with TransactGetItems (one snapshot). When pagination is requested the query starts after the EntityIdToken id (SK > CHILD#pad(lastId)) and the page is built with Page.mapped, as relational-jdbc does. Pages are read lazily and stop once the page is full. Each request reads at most pageSize + 1 records; the bound is per request, not per stream, so entityFilter, which runs after the read, cannot leave a page short.
  • A concrete subtype is a single partition. ANY_SUBTYPE reads each subtype partition of the type, capping each at pageSize + 1 ids, and merges them by id. The merged stream stays in id order, so a single EntityIdToken pages across subtypes. The subtype set is derived from PolarisEntitySubType by parent type, so a new subtype is picked up automatically.
  • hasChildren: a Limit 1, strongly consistent query per (type, subtype) partition, stopping at the first hit. With no type given, it checks each entity type in turn; NULL_TYPE and ROOT are skipped, since they are never stored as children. The subtype set is a fixed enum, so the fan-out is bounded (only TABLE_LIKE has more than 0).
  • Drop gsi1 entirely: listing no longer needs an index, so remove it from the schema (DynamoDbSchema), drop the GSI1* constants and the childrenGsi* encoders from DynamoDbLayout, and stop writing gsi1pk/gsi1sk on the entity item (EntityItem).

Consistency

BasePersistence asks reads for a consistent view. All reads here meet that: lookups by id and by name, and now listing and hasChildren, use strongly consistent base-table reads. A child written just before a list is always seen, because its child record is written in the same TransactWriteItems as the entity.

Making these checks atomic with the following mutation (the drop/overlap TODO in AtomicOperationMetaStoreManager) remains a separate concurrency concern, unchanged by this PR and the same for every backend.

Testing

  • DynamoDbListingIT: id order without pagination, paging over several children with no gaps or repeats, a subtype filter that drops items mid-query, listFullEntities filter and transformer across pages, an empty parent, hasChildren with and without a type, a deleted child no longer listed, and another realm's children in the same table not listed.
  • DynamoDbSchemaIT: the table is provisioned with no secondary index.
  • DynamoDbLayoutTest, EntityItemTest: updated for the removed GSI encoders/attributes.
  • ./gradlew :polaris-persistence-dynamodb-atomic:spotlessCheck compileIntTestJava test passes locally. The ITs need Docker and run in CI.

Checklist

@github-project-automation github-project-automation Bot moved this to PRs In Progress in Basic Kanban Board Nov 9, 2026
@ZephyrYWZhou
ZephyrYWZhou force-pushed the dynamodb-atomic-listing branch from 4356dee to a393bda Compare November 12, 2026 19:44
@ZephyrYWZhou
ZephyrYWZhou marked this pull request as ready for review November 13, 2026 06:42

@orca-review-bot orca-review-bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice, clean step. Paging matches JDBC, the lazy paginator plus Page.mapped means no gaps or repeats, and the consistency write-up is honest. Two small inline things below. Unanchored test asks: (1) can we add a realm-isolation case to DynamoDbListingIT, where a second persistence with another realmId writes into the same table and parent and isn't listed? (2) can we add a short comment there that the list-right-after-write assertions depend on the Floci emulator updating the GSI immediately? Agree that no CHANGELOG entry is needed since the module isn't wired up yet.

🤖 AI-generated review comment — agents make mistakes, so please verify before acting on it.

@ZephyrYWZhou
ZephyrYWZhou force-pushed the dynamodb-atomic-listing branch from fc307de to fa5a7e5 Compare November 14, 2026 00:28

@flyrain flyrain left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks a lot for working on it, @ZephyrYWZhou !
Does GSI provide strong consistency? If not, I’m concerned that GSI lag affects correctness. For example, a table creation can succeed, then a subsequent namespace drop can miss that table and incorrectly treat the namespace as empty. This can happen with sequential requests, so it goes beyond the existing concurrency TODO. Location overlap validation also relies on listing and could miss a newly created sibling.

Scanning all entities wont' be scalable at the same time when it comes to list. Wondering if the following solution makes sense. We can store children records in the base table, keyed by realm, catalog, parent ID and entity type, with child ID as the sort key? Maintaining these records in the same transaction as the entity changes would let us use efficient queries with ConsistentRead=true.

This adds some write and maintenance overhead, but ensures these checks see writes that already completed. Making the checks atomic with subsequent mutations would remain a separate concurrency concern.

WDYT?

@ZephyrYWZhou

ZephyrYWZhou commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor Author

Thanks a lot for working on it, @ZephyrYWZhou ! Does GSI provide strong consistency? If not, I’m concerned that GSI lag affects correctness. For example, a table creation can succeed, then a subsequent namespace drop can miss that table and incorrectly treat the namespace as empty. This can happen with sequential requests, so it goes beyond the existing concurrency TODO. Location overlap validation also relies on listing and could miss a newly created sibling.

Scanning all entities wont' be scalable at the same time when it comes to list. Wondering if the following solution makes sense. We can store children records in the base table, keyed by realm, catalog, parent ID and entity type, with child ID as the sort key? Maintaining these records in the same transaction as the entity changes would let us use efficient queries with ConsistentRead=true.

This adds some write and maintenance overhead, but ensures these checks see writes that already completed. Making the checks atomic with subsequent mutations would remain a separate concurrency concern.

WDYT?

Thanks @flyrain for the deep dive insights and this is a correctness gap I agree with you and am willing to fix.

You are correct that GSI is only eventually consistent and not strongly consistent on read. So the drop-namespace emptiness check and the location-overlap check can indeed read a stale index and decide wrong, even for sequential requests.

I'll move children into the base DDB table as you suggest: an item keyed by realm + catalog + parentId + entityType with childId as the sort key, written in the same TransactWriteItems as the entity on create, delete, and rename (the create/rename/delete paths are already transactional and already move the name sentinel, so this rides along the same transaction). Listing then becomes a base-table Query with ConsistentRead=true. We can then trust a strict consistency there.

I will remove gsi1 entirely as well.

For a cost analysis, dropping gsi1 for a base-table child item keeps write and storage cost about the same, makes list reads a bit pricier (strong reads cost 2× read units), and in return removes the GSI-lag correctness bug and the risk of full-table scans.

Another technical cost is one extra item per entity and one more action per write transaction (which also lowers the per-transaction entity count against the 100-action limit), but I think the correctness is worth it.

I'll update the PR along these lines.

@ZephyrYWZhou

Copy link
Copy Markdown
Contributor Author

One more thought on sequencing @flyrain. Since this change touches the write paths, not just listing, so I'd like to split it into two PRs: I'll put up a separate PR first to write the child record in the base table on create/rename/delete, same TransactWriteItems as the entity; gsi1 and listing stay as-is, so nothing breaks. Once that's in, I'll route back to this PR to switch listEntities/hasChildren to a base-table Query and remove gsi1. WDYT?

@flyrain

flyrain commented Nov 17, 2026

Copy link
Copy Markdown
Contributor

Two PRs make sense to me. Thanks!

@ZephyrYWZhou

Copy link
Copy Markdown
Contributor Author

Sounds good! Opened #6118 as the first PR to write child records in the base table. I'll come back to this one for the listing switch and gsi1 removal once it's in.

…ords

List a parent's children by a strongly consistent base-table Query on the
child records (realm#catalog#parent#type#subtype, child id in the sort key),
then read the ids into entities with TransactGetItems. Paging uses
EntityIdToken, as relational-jdbc does. A concrete subtype is one partition;
ANY_SUBTYPE reads each subtype partition and concatenates, for read-everything
callers only (paged ANY_SUBTYPE is rejected).

Drop gsi1 entirely: listing no longer needs an index, so remove the index from
the schema and stop writing gsi1pk/gsi1sk on the entity item. This removes the
GSI-lag correctness gap flagged on the original listing PR, since the child
records are maintained in the entity's own transaction.

Builds on the child records added in apache#6118.
@ZephyrYWZhou
ZephyrYWZhou force-pushed the dynamodb-atomic-listing branch from fa5a7e5 to c4a382b Compare November 18, 2026 02:29
The ported IT pages ANY_SUBTYPE (listFullEntities with a limit), so rejecting
it broke two cases. Instead, read each subtype partition and merge by id: each
partition is id-ordered and resumes after the page token, so the merged stream
stays id-ordered and a single EntityIdToken pages across subtypes.
@ZephyrYWZhou

ZephyrYWZhou commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor Author

@flyrain this PR is reworked and ready for another look!

Listing and hasChildren now read the child records from #6118 with ConsistentRead=true instead of gsi1.

One note: since the subtype is now in the partition key, an ANY_SUBTYPE listing spans several partitions, so it reads each one and merges them by id. That keeps the results in id order and lets a single page token resume across all of them.

@orca-review-bot orca-review-bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice move off the GSI. Strongly consistent child-record reads are the right call for emptiness and overlap checks. A few things:

  • The ANY_SUBTYPE merge reads every partition in full on every page, and entities come in one GetItem at a time. Inline comments have the details.
  • Tests: can we add a paged ANY_SUBTYPE IT with interleaved table/view/generic ids? Today every ANY_SUBTYPE test only writes ICEBERG_TABLE, so the merge never interleaves. Can we also add a hasChildren(null, ...) case where the only child is a nested namespace?
  • The PR description still says paged ANY_SUBTYPE throws UnsupportedOperationException, and it says entities are read with TransactGetItems. Can we update both to match the head commit?
  • Realm isolation looks good: keys are realm-prefixed and there's a test for it. No changelog needed since the module isn't wired in yet.

🤖 AI-generated review comment — agents make mistakes, so please verify before acting on it.

- Cap each subtype partition at pageSize + 1 ids in the ANY_SUBTYPE merge, so a
  paged merged listing no longer drains whole partitions.
- Read each page's ids with batchReadEntities (TransactGetItems, one snapshot
  per chunk) instead of a GetItem per id, matching the javadoc.
- Derive the child subtypes from PolarisEntitySubType.values() by parent type,
  so a newly added subtype is listed automatically instead of being skipped.
- Use Objects::nonNull.
- Tests: paged ANY_SUBTYPE with interleaved table/view/generic ids, and
  hasChildren(null, ...) whose only child is a nested namespace.
… a page

The previous commit capped each subtype stream with a Java .limit(pageSize+1)
and collected eagerly before filtering. That truncated the stream, so
listFullEntities' filter could not pull past the cap and the page came back
short (CI: listFullEntitiesAppliesTheFilterAndTransformer).

Keep the per-request DynamoDB Limit for efficiency but not the stream .limit:
the paginator keeps fetching as the caller consumes. ANY_SUBTYPE now does a
lazy k-way merge of the per-subtype id streams by id, and each id is read into
its entity with a strongly consistent get (reverting the eager batch read, so
the stream stays lazy end to end).
@ZephyrYWZhou
ZephyrYWZhou requested a review from flyrain November 18, 2026 17:56
}
// Read each id into its entity with a strongly consistent get, lazily, so the stream keeps
// producing while a downstream filter pulls more.
return ids.map(this::readEntity).filter(Objects::nonNull);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

First, the child query and the entity read are separate requests, and we don't check that the entity still belongs to this parent. If a rename moves the entity to another namespace in between, it gets listed under the old parent, carrying its new parentId. lookupEntityByName already guards against this with a catalogId/parentId/type check after readEntity. Can we do the same here, plus a subtype check when it's concrete? Missing a child that moved in after the query is fine, since that's just a later write. But returning one that moved out is wrong data.

Second, this does one GetItem per entity. A 100-item page means 100 round trips, one after another. I think the earlier batch version broke because of collecting everything up front, not because of the batching itself. Could we read the ids in chunks of pageSize + 1 through batchReadEntities and flatMap over the chunks? That stays lazy, so the filter can still pull more, and it's one round trip per chunk instead of per entity.

Could we also add an IT that moves a table to another namespace and checks it's gone from the old one?

@ZephyrYWZhou ZephyrYWZhou Nov 18, 2026 •

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see I see fair enough, the rename gap is real, let me add the guard to it.

For the second point, yeah exactly, I was originally collecting all IDs together and do batchReadEntities, which breaks laziness. After which it is reverted to 100 ID -> 100 GetItems call mode. Didn't think of solution using flatMap, let me do that. Thanks!

- Re-check each listed entity against the queried catalog/parent/type (and
  subtype when concrete), so a child re-parented between the child-record query
  and the entity read is not returned under its old parent. Mirrors the guard in
  lookupEntityByName.
- Read ids a chunk at a time (pageSize + 1, capped at the TransactGetItems item
  limit) via batchReadEntities, flat-mapped so the stream stays lazy: one round
  trip per chunk instead of one GetItem per entity, and the filter can still pull
  more. The earlier batch attempt broke from collecting all ids up front, not
  from batching.
- IT: move a table to another namespace and assert it leaves the old one.

@flyrain flyrain left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1 Thanks a lot for working on it, @ZephyrYWZhou !

@github-project-automation github-project-automation Bot moved this from PRs In Progress to Ready to merge in Basic Kanban Board Nov 18, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants