Repository navigation
Conversation
|
#5820 also addresses #5807. It adds each requirement-only table to the commit as an update, so the table's entity is rewritten with a new version. This PR instead checks such tables without writing them, atomically with the other tables' updates, so that:
|
orca-review-bot
left a comment
There was a problem hiding this comment.
Nice fix. The per-backend checks look right, sorting the JDBC locks is a good call, and the race tests are thorough. Main things: (1) this adds abstract methods to PolarisMetaStoreManager and BasePersistence, so custom backends break. Can we link the dev-list discussion, and settle this vs #5820? (2) There's one JDBC retry path that can still return a 500 on a conflict (inline). (3) NoSQL doesn't always return the failure status the javadoc promises (inline). CHANGELOG and authz look fine.
🤖 AI-generated review comment — agents make mistakes, so please verify before acting on it.
| private void checkEntityUnchanged(Connection connection, PolarisEntityCore expected) | ||
| throws SQLException { | ||
| AtomicReference<PolarisChangeTrackingVersions> current = new AtomicReference<>(); | ||
| datasourceOperations.executeSelectOverStream( |
There was a problem hiding this comment.
This select runs through withRetries, which retries 40001 on the same connection. Inside runWithinTransaction that connection's transaction is already aborted, so the retry fails with a different SQLState. The 40001/40P01 mapping below doesn't catch that, and the caller gets a 500. It's harmless with the default max-retries=1, and Postgres at READ COMMITTED won't raise 40001 here. But CockroachDB with max-retries=3 (the docs example) would hit it. Can we skip the inner retry when we're already inside the transaction?
🤖 AI-generated review comment — agents make mistakes, so please verify before acting on it.
| var result = all.get(i); | ||
| if (result.getReturnStatus() != BaseResult.ReturnStatus.SUCCESS) { | ||
| var status = | ||
| result.getReturnStatus() == BaseResult.ReturnStatus.ENTITY_NOT_FOUND |
There was a problem hiding this comment.
Only ENTITY_NOT_FOUND is remapped here, so any other failed status passes through as-is. The writeEntitiesIfConditionsHold javadoc says every failed condition or write gives TARGET_ENTITY_CONCURRENTLY_MODIFIED, and the transactional impl always returns that. Can we map every non-success status here, or loosen the javadoc?
🤖 AI-generated review comment — agents make mistakes, so please verify before acting on it.
| * @param batch the conditions to check and the writes to apply | ||
| * @return the written entities in the order of the batch's writes, or a failure | ||
| */ | ||
| @NonNull ConditionalEntityWriteResult writeEntitiesIfConditionsHold( |
There was a problem hiding this comment.
New abstract SPI method (plus BasePersistence.checkAndWriteEntities), so out-of-tree backends stop compiling. AGENTS.md asks for that to go through the dev list. Is there a thread we can link, especially since #5820 fixes the same bug without changing the SPI?
🤖 AI-generated review comment — agents make mistakes, so please verify before acting on it.
There was a problem hiding this comment.
We did have a dev thread, https://lists.apache.org/thread.html/mkkmwfxr77qkzn5yodwdrjcknss7o4xy.
4b351df to
f9d7f1c
Compare
kotwal-itpro
left a comment
There was a problem hiding this comment.
Thanks for this, @qlong. I read through the SPI, the JDBC and NoSQL implementations and the handler change, but haven't run it locally.
The handler side looks right to me. Comparing the resolved entity's metadata location with the metadata the requirements were validated against closes the window between loadTable and the resolve, and the atomic Unchanged check covers everything after that. testRequirementOnlyTableCommittedBetweenItsLoadAndItsResolveFailsTheCommit pins the first part.
One question on the JDBC side. checkEntityUnchanged takes SELECT ... FOR UPDATE on a row it only checks. The result stays correct, but two commits that only check the same table now wait for each other. They don't fail, as testBatchesThatOnlyCheckTheSameEntityDoNotConflict shows, but they run one at a time. When many commits assert the same table, which is the case Yufei raised on the list, they would queue on that one row.
FOR SHARE would let commits that only check the row run in parallel, and still make a concurrent update of the row wait until the batch commits. The (catalog_id, id) lock order would still rule out deadlocks, since a batch either checks or writes an entity, never both. PostgreSQL supports FOR SHARE. I'm not sure about H2, and I think CockroachDB only takes shared locks under READ COMMITTED by default, so this may need to depend on the database type. Is FOR UPDATE deliberate here, or would FOR SHARE work?
For context, the multi-table promotion doc in #5740 relies on the multi-table commit being atomic, so I'm glad to see #5807 fixed.
f9d7f1c to
fc23f9a
Compare
|
Thanks @qlong, |
iprithv
left a comment
There was a problem hiding this comment.
I think this makes sense, checkEntityUnchanged goes through executeSelectOverStream(connection, …), and that overload still runs withRetries on the same connection.
We’re already inside runWithinTransaction here. If the SELECT hits 40001 (easy to see on Cockroach with max-retries>1, the docs example uses 3), the transaction is aborted, the retry gets something like 25P02 instead, and the 40001/40P01 mapping below never sees the original conflict. Caller ends up with a 500 instead of a concurrency failure / 409.
persistEntity’s execute(connection) path doesn’t retry, which is what we want inside a txn. Can we do the same for this select, skip the inner retry when we already own the connection?
|
|
||
| /** {@inheritDoc} */ | ||
| @Override | ||
| public @NonNull EntitiesResult updateEntitiesPropertiesIfNotChanged( |
There was a problem hiding this comment.
Can we deprecate it instead of replacing it with a new method?
|
Thanks @qlong this reads well! Reducing the multi-table commit race to one general |
| Set<EntityKey> entities = new HashSet<>(); | ||
| for (Write write : writes) { | ||
| if (!(write instanceof Write.Update update)) { | ||
| throw new IllegalArgumentException("Unsupported write " + write); |
There was a problem hiding this comment.
Since Write and Condition are sealed, could we use a switch over the sealed type in these places (here, AtomicOperationMetaStoreManager, JDBC, NoSQL, transactional)? Adding a new kind would then fail to compile in every backend, rather than throw IllegalArgumentException at run time. WDYT?
There was a problem hiding this comment.
Good point on error out at build time. polaris-core is jdk-17 so we cannot do exhaustive switch here. I will change jdbc / nosql backend to use switch for now
| if (!(write instanceof Write.Update update)) { | ||
| throw new IllegalArgumentException("Unsupported write " + write); | ||
| } | ||
| requireOnce(entities, update.entity()); |
There was a problem hiding this comment.
If a multi-table commit lists the same table twice, once with updates and once requirement-only, this throws IllegalArgumentException. Does the REST layer reject duplicate identifiers before this point? If not, I think the client sees a 500 instead of a 4xx. Could we add a test for it?
There was a problem hiding this comment.
Agreed that it should be a 400 bad request in that case.
There was a problem hiding this comment.
commitTransaction group changes by table Id, so this probably won't happen but I think it is a good defensive coding to have. Will change to throw 400 if a condition write cannot be built
+1. Will remove retry within transaction. I left DatasourceOperations.executeSelectOverStream(connection, …) as it is to not extend the scope of this PR, will fix it in follow up. |
fc23f9a to
8f10c69
Compare
|
Heads-up: #5967 has been merged, so Could you rebase onto |
Add conditional entity writes to the metastore SPI: a set of entity writes applied atomically, and only if conditions on other entities hold. Until now the only atomic condition a write could express was "the entity being written is still at its version", so a decision based on reading other entities could not be enforced at write time. This adds the general primitive; this change uses it to fix apache#5807, and later condition and write kinds (for example parent checks, create, drop) can be added without changing the SPI. Persistence: - Add `PolarisMetaStoreManager.writeEntitiesIfConditionsHold(ConditionalEntityWrite)`. `Condition.Unchanged` checks an entity's version without writing it; `Write.Update` is the existing compare-and-swap update. If any condition or write fails, nothing is written and the result is `TARGET_ENTITY_CONCURRENTLY_MODIFIED`. - Backends apply it through the new `BasePersistence.checkAndWriteEntities(conditions, changes)`. Conditions, writes and entity changes are sealed types at both layers, so later kinds (such as create and drop) are added without new methods. - `updateEntitiesPropertiesIfNotChanged` is now a default method that delegates to it. - Implement it on all backends. JDBC checks a row with `SELECT ... FOR SHARE` on PostgreSQL (`FOR UPDATE` on H2 and CockroachDB, which lack shared row locks by default) in the same transaction as the updates, takes row locks in (catalog_id, id) order, and reports deadlocks (40P01) and serialization failures (40001) as conflicts. NoSQL adds a `CHECK` mutation, evaluated in the same commit and again on retry. The DynamoDB backend rejects it, as it does its other not-yet-implemented multi-entity writes. Fix (apache#5807): - A multi-table commit can list a table with requirements but no updates, to assert it is unchanged. Such tables were validated when the commit was staged but were not part of the atomic write, so two concurrent commits that each changed one table and asserted the other could both succeed. - `commitTransaction` now adds each requirement-only table as an `Unchanged` condition, after verifying that its entity still points to the metadata the requirements were validated against. Commits that only assert the same table do not fail each other. Tests: conformance tests on all four backends (including PostgreSQL), with concurrent writers for write skew and crossed updates; a NoSQL test that forces a retry after a failed compare-and-swap; handler tests that inject a concurrent commit at the two points of the race. Fixes apache#5807
- JDBC: run the check's select once inside the transaction, so that a serialization failure reaches the conflict mapping instead of being retried within the aborted transaction. - Deprecate updateEntitiesPropertiesIfNotChanged in favour of writeEntitiesIfConditionsHold. - JDBC and NoSQL: exhaustive switch over the sealed condition and write kinds, so a new kind fails to compile there. - commitTransaction: reject a conditional write that cannot be built with a 400, and clean up the metadata files already written when the commit fails before the conditional write is applied.
CHANGELOG.md is no longer edited directly since apache#5967. Revert the entries added to it and add them as a fragment instead, with a new Deprecations entry for updateEntitiesPropertiesIfNotChanged.
8f10c69 to
88fa024
Compare
Added a fragment for SPI change. |
Add conditional entity writes to the metastore SPI: a set of entity writes applied atomically, and only if conditions on other entities hold. Until now the only atomic condition a write could express was "the entity being written is still at its version", so a decision based on reading other entities could not be enforced at write time. This adds the general primitive; this change uses it to fix #5807, and later condition and write kinds (for example parent checks, create, drop) can be added without changing the SPI.
Persistence:
PolarisMetaStoreManager.writeEntitiesIfConditionsHold(ConditionalEntityWrite).Condition.Unchangedchecks an entity's version without writing it;Write.Updateis the existing compare-and-swap update. If any condition or write fails, nothing is written and the result isTARGET_ENTITY_CONCURRENTLY_MODIFIED.updateEntitiesPropertiesIfNotChangedis now a default method that delegates to it.SELECT ... FOR UPDATEin the same transaction as the updates, takes row locks in (catalog_id, id) order, and reports deadlocks (40P01) and serialization failures (40001) as conflicts. NoSQL adds aCHECKmutation, evaluated in the same commit and again on retry.Fix (#5807):
commitTransactionnow adds each requirement-only table as anUnchangedcondition, after verifying that its entity still points to the metadata the requirements were validated against. Commits that only assert the same table do not fail each other.Tests: conformance tests on all four backends (including PostgreSQL), with concurrent writers for write skew and crossed updates; a NoSQL test that forces a retry after a failed compare-and-swap; handler tests that inject a concurrent commit at the two points of the race.
Fixes #5807
Dev ML: https://lists.apache.org/thread.html/mkkmwfxr77qkzn5yodwdrjcknss7o4xy
Checklist
CHANGELOG.md(if needed)site/content/in-dev/unreleased(if needed)