Repository navigation
Persistence: DynamoDB (atomic) listing and hasChildren - #6016
ZephyrYWZhou wants to merge 5 commits into
Conversation
4356dee to
a393bda
Compare
orca-review-bot
left a comment
There was a problem hiding this comment.
Nice, clean step. Paging matches JDBC, the lazy paginator plus Page.mapped means no gaps or repeats, and the consistency write-up is honest. Two small inline things below. Unanchored test asks: (1) can we add a realm-isolation case to DynamoDbListingIT, where a second persistence with another realmId writes into the same table and parent and isn't listed? (2) can we add a short comment there that the list-right-after-write assertions depend on the Floci emulator updating the GSI immediately? Agree that no CHANGELOG entry is needed since the module isn't wired up yet.
🤖 AI-generated review comment — agents make mistakes, so please verify before acting on it.
fc307de to
fa5a7e5
Compare
flyrain
left a comment
There was a problem hiding this comment.
Thanks a lot for working on it, @ZephyrYWZhou !
Does GSI provide strong consistency? If not, I’m concerned that GSI lag affects correctness. For example, a table creation can succeed, then a subsequent namespace drop can miss that table and incorrectly treat the namespace as empty. This can happen with sequential requests, so it goes beyond the existing concurrency TODO. Location overlap validation also relies on listing and could miss a newly created sibling.
Scanning all entities wont' be scalable at the same time when it comes to list. Wondering if the following solution makes sense. We can store children records in the base table, keyed by realm, catalog, parent ID and entity type, with child ID as the sort key? Maintaining these records in the same transaction as the entity changes would let us use efficient queries with ConsistentRead=true.
This adds some write and maintenance overhead, but ensures these checks see writes that already completed. Making the checks atomic with subsequent mutations would remain a separate concurrency concern.
WDYT?
Thanks @flyrain for the deep dive insights and this is a correctness gap I agree with you and am willing to fix. You are correct that GSI is only eventually consistent and not strongly consistent on read. So the drop-namespace emptiness check and the location-overlap check can indeed read a stale index and decide wrong, even for sequential requests. I'll move children into the base DDB table as you suggest: an item keyed by I will remove For a cost analysis, dropping Another technical cost is one extra item per entity and one more action per write transaction (which also lowers the per-transaction entity count against the 100-action limit), but I think the correctness is worth it. I'll update the PR along these lines. |
|
One more thought on sequencing @flyrain. Since this change touches the write paths, not just listing, so I'd like to split it into two PRs: I'll put up a separate PR first to write the child record in the base table on |
|
Two PRs make sense to me. Thanks! |
|
Sounds good! Opened #6118 as the first PR to write child records in the base table. I'll come back to this one for the listing switch and gsi1 removal once it's in. |
…ords List a parent's children by a strongly consistent base-table Query on the child records (realm#catalog#parent#type#subtype, child id in the sort key), then read the ids into entities with TransactGetItems. Paging uses EntityIdToken, as relational-jdbc does. A concrete subtype is one partition; ANY_SUBTYPE reads each subtype partition and concatenates, for read-everything callers only (paged ANY_SUBTYPE is rejected). Drop gsi1 entirely: listing no longer needs an index, so remove the index from the schema and stop writing gsi1pk/gsi1sk on the entity item. This removes the GSI-lag correctness gap flagged on the original listing PR, since the child records are maintained in the entity's own transaction. Builds on the child records added in apache#6118.
fa5a7e5 to
c4a382b
Compare
The ported IT pages ANY_SUBTYPE (listFullEntities with a limit), so rejecting it broke two cases. Instead, read each subtype partition and merge by id: each partition is id-ordered and resumes after the page token, so the merged stream stays id-ordered and a single EntityIdToken pages across subtypes.
|
@flyrain this PR is reworked and ready for another look! Listing and One note: since the |
orca-review-bot
left a comment
There was a problem hiding this comment.
Nice move off the GSI. Strongly consistent child-record reads are the right call for emptiness and overlap checks. A few things:
- The
ANY_SUBTYPEmerge reads every partition in full on every page, and entities come in one GetItem at a time. Inline comments have the details. - Tests: can we add a paged
ANY_SUBTYPEIT with interleaved table/view/generic ids? Today everyANY_SUBTYPEtest only writesICEBERG_TABLE, so the merge never interleaves. Can we also add ahasChildren(null, ...)case where the only child is a nested namespace? - The PR description still says paged
ANY_SUBTYPEthrowsUnsupportedOperationException, and it says entities are read withTransactGetItems. Can we update both to match the head commit? - Realm isolation looks good: keys are realm-prefixed and there's a test for it. No changelog needed since the module isn't wired in yet.
🤖 AI-generated review comment — agents make mistakes, so please verify before acting on it.
- Cap each subtype partition at pageSize + 1 ids in the ANY_SUBTYPE merge, so a paged merged listing no longer drains whole partitions. - Read each page's ids with batchReadEntities (TransactGetItems, one snapshot per chunk) instead of a GetItem per id, matching the javadoc. - Derive the child subtypes from PolarisEntitySubType.values() by parent type, so a newly added subtype is listed automatically instead of being skipped. - Use Objects::nonNull. - Tests: paged ANY_SUBTYPE with interleaved table/view/generic ids, and hasChildren(null, ...) whose only child is a nested namespace.
… a page The previous commit capped each subtype stream with a Java .limit(pageSize+1) and collected eagerly before filtering. That truncated the stream, so listFullEntities' filter could not pull past the cap and the page came back short (CI: listFullEntitiesAppliesTheFilterAndTransformer). Keep the per-request DynamoDB Limit for efficiency but not the stream .limit: the paginator keeps fetching as the caller consumes. ANY_SUBTYPE now does a lazy k-way merge of the per-subtype id streams by id, and each id is read into its entity with a strongly consistent get (reverting the eager batch read, so the stream stays lazy end to end).
| } | ||
| // Read each id into its entity with a strongly consistent get, lazily, so the stream keeps | ||
| // producing while a downstream filter pulls more. | ||
| return ids.map(this::readEntity).filter(Objects::nonNull); |
There was a problem hiding this comment.
First, the child query and the entity read are separate requests, and we don't check that the entity still belongs to this parent. If a rename moves the entity to another namespace in between, it gets listed under the old parent, carrying its new parentId. lookupEntityByName already guards against this with a catalogId/parentId/type check after readEntity. Can we do the same here, plus a subtype check when it's concrete? Missing a child that moved in after the query is fine, since that's just a later write. But returning one that moved out is wrong data.
Second, this does one GetItem per entity. A 100-item page means 100 round trips, one after another. I think the earlier batch version broke because of collecting everything up front, not because of the batching itself. Could we read the ids in chunks of pageSize + 1 through batchReadEntities and flatMap over the chunks? That stays lazy, so the filter can still pull more, and it's one round trip per chunk instead of per entity.
Could we also add an IT that moves a table to another namespace and checks it's gone from the old one?
There was a problem hiding this comment.
I see I see fair enough, the rename gap is real, let me add the guard to it.
For the second point, yeah exactly, I was originally collecting all IDs together and do batchReadEntities, which breaks laziness. After which it is reverted to 100 ID -> 100 GetItems call mode. Didn't think of solution using flatMap, let me do that. Thanks!
- Re-check each listed entity against the queried catalog/parent/type (and subtype when concrete), so a child re-parented between the child-record query and the entity read is not returned under its old parent. Mirrors the guard in lookupEntityByName. - Read ids a chunk at a time (pageSize + 1, capped at the TransactGetItems item limit) via batchReadEntities, flat-mapped so the stream stays lazy: one round trip per chunk instead of one GetItem per entity, and the filter can still pull more. The earlier batch attempt broke from collecting all ids up front, not from batching. - IT: move a table to another namespace and assert it leaves the old one.
flyrain
left a comment
There was a problem hiding this comment.
+1 Thanks a lot for working on it, @ZephyrYWZhou !
Summary
Next step of the DynamoDB (atomic) backend: listing. This implements
listEntities,listFullEntitiesandhasChildrenonDynamoDbBasePersistence, reading the strongly consistent child records on the base table.Builds on the child records added in #6118, now merged. It replaces the earlier
gsi1-based approach from the review discussion: a GSI is eventually consistent, so listing it could make a wrong correctness decision (drop-namespace emptiness, location-overlap) even for sequential requests. Child records live on the base table and are maintained in the entity's own transaction, so they can be read withConsistentRead=true.Changes
listEntities/listFullEntities: query the child records for one parent, type and subtype (PK = realmId#catalogId#parentId#typeCode#subTypeCode), in id order, withConsistentRead=true. The child records hold only ids, so each page's ids are then read into entities withTransactGetItems(one snapshot). When pagination is requested the query starts after theEntityIdTokenid (SK > CHILD#pad(lastId)) and the page is built withPage.mapped, asrelational-jdbcdoes. Pages are read lazily and stop once the page is full. Each request reads at mostpageSize + 1records; the bound is per request, not per stream, soentityFilter, which runs after the read, cannot leave a page short.ANY_SUBTYPEreads each subtype partition of the type, capping each atpageSize + 1ids, and merges them by id. The merged stream stays in id order, so a singleEntityIdTokenpages across subtypes. The subtype set is derived fromPolarisEntitySubTypeby parent type, so a new subtype is picked up automatically.hasChildren: aLimit 1, strongly consistent query per (type, subtype) partition, stopping at the first hit. With no type given, it checks each entity type in turn;NULL_TYPEandROOTare skipped, since they are never stored as children. The subtype set is a fixed enum, so the fan-out is bounded (onlyTABLE_LIKEhas more than0).gsi1entirely: listing no longer needs an index, so remove it from the schema (DynamoDbSchema), drop theGSI1*constants and thechildrenGsi*encoders fromDynamoDbLayout, and stop writinggsi1pk/gsi1skon the entity item (EntityItem).Consistency
BasePersistenceasks reads for a consistent view. All reads here meet that: lookups by id and by name, and now listing andhasChildren, use strongly consistent base-table reads. A child written just before a list is always seen, because its child record is written in the sameTransactWriteItemsas the entity.Making these checks atomic with the following mutation (the drop/overlap TODO in
AtomicOperationMetaStoreManager) remains a separate concurrency concern, unchanged by this PR and the same for every backend.Testing
DynamoDbListingIT: id order without pagination, paging over several children with no gaps or repeats, a subtype filter that drops items mid-query,listFullEntitiesfilter and transformer across pages, an empty parent,hasChildrenwith and without a type, a deleted child no longer listed, and another realm's children in the same table not listed.DynamoDbSchemaIT: the table is provisioned with no secondary index.DynamoDbLayoutTest,EntityItemTest: updated for the removed GSI encoders/attributes../gradlew :polaris-persistence-dynamodb-atomic:spotlessCheck compileIntTestJava testpasses locally. The ITs need Docker and run in CI.Checklist