Skip to content

Fix multi-table commit write skew using conditional entity writes - #5961

Open
qlong wants to merge 3 commits into
apache:mainfrom
qlong:fix-5807-requirement-only-table-changes
Open

qlong wants to merge 3 commits into
apache:mainfrom
qlong:fix-5807-requirement-only-table-changes

Conversation

@qlong

@qlong qlong commented Nov 2, 2026 •

Copy link
Copy Markdown

Add conditional entity writes to the metastore SPI: a set of entity writes applied atomically, and only if conditions on other entities hold. Until now the only atomic condition a write could express was "the entity being written is still at its version", so a decision based on reading other entities could not be enforced at write time. This adds the general primitive; this change uses it to fix #5807, and later condition and write kinds (for example parent checks, create, drop) can be added without changing the SPI.

Persistence:

  • Add PolarisMetaStoreManager.writeEntitiesIfConditionsHold(ConditionalEntityWrite). Condition.Unchanged checks an entity's version without writing it; Write.Update is the existing compare-and-swap update. If any condition or write fails, nothing is written and the result is TARGET_ENTITY_CONCURRENTLY_MODIFIED.
  • updateEntitiesPropertiesIfNotChanged is now a default method that delegates to it.
  • Implement it on all backends. JDBC checks with SELECT ... FOR UPDATE in the same transaction as the updates, takes row locks in (catalog_id, id) order, and reports deadlocks (40P01) and serialization failures (40001) as conflicts. NoSQL adds a CHECK mutation, evaluated in the same commit and again on retry.

Fix (#5807):

  • A multi-table commit can list a table with requirements but no updates, to assert it is unchanged. Such tables were validated when the commit was staged but were not part of the atomic write, so two concurrent commits that each changed one table and asserted the other could both succeed.
  • commitTransaction now adds each requirement-only table as an Unchanged condition, after verifying that its entity still points to the metadata the requirements were validated against. Commits that only assert the same table do not fail each other.

Tests: conformance tests on all four backends (including PostgreSQL), with concurrent writers for write skew and crossed updates; a NoSQL test that forces a retry after a failed compare-and-swap; handler tests that inject a concurrent commit at the two points of the race.

Fixes #5807

Dev ML: https://lists.apache.org/thread.html/mkkmwfxr77qkzn5yodwdrjcknss7o4xy

Checklist

  • 🛡️ Don't disclose security issues! (contact security@apache.org)
  • 🔗 Clearly explained why the changes are needed, or linked related issues: Fixes #
  • 🧪 Added/updated tests with good coverage, or manually tested (and explained how)
  • 💡 Added comments for complex logic
  • 🧾 Updated CHANGELOG.md (if needed)
  • 📚 Updated documentation in site/content/in-dev/unreleased (if needed)

@qlong

qlong commented Nov 2, 2026

Copy link
Copy Markdown
Author

#5820 also addresses #5807. It adds each requirement-only table to the commit as an update, so the table's entity is rewritten with a new version. This PR instead checks such tables without writing them, atomically with the other tables' updates, so that:

  • commits that only assert the same table don't conflict with each other, and the table's version isn't bumped;
  • the check is bound to the metadata the requirements were validated against, so a commit between loading and resolving the table is detected.

@orca-review-bot orca-review-bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice fix. The per-backend checks look right, sorting the JDBC locks is a good call, and the race tests are thorough. Main things: (1) this adds abstract methods to PolarisMetaStoreManager and BasePersistence, so custom backends break. Can we link the dev-list discussion, and settle this vs #5820? (2) There's one JDBC retry path that can still return a 500 on a conflict (inline). (3) NoSQL doesn't always return the failure status the javadoc promises (inline). CHANGELOG and authz look fine.

🤖 AI-generated review comment — agents make mistakes, so please verify before acting on it.

private void checkEntityUnchanged(Connection connection, PolarisEntityCore expected)
throws SQLException {
AtomicReference<PolarisChangeTrackingVersions> current = new AtomicReference<>();
datasourceOperations.executeSelectOverStream(

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This select runs through withRetries, which retries 40001 on the same connection. Inside runWithinTransaction that connection's transaction is already aborted, so the retry fails with a different SQLState. The 40001/40P01 mapping below doesn't catch that, and the caller gets a 500. It's harmless with the default max-retries=1, and Postgres at READ COMMITTED won't raise 40001 here. But CockroachDB with max-retries=3 (the docs example) would hit it. Can we skip the inner retry when we're already inside the transaction?

🤖 AI-generated review comment — agents make mistakes, so please verify before acting on it.

var result = all.get(i);
if (result.getReturnStatus() != BaseResult.ReturnStatus.SUCCESS) {
var status =
result.getReturnStatus() == BaseResult.ReturnStatus.ENTITY_NOT_FOUND

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Only ENTITY_NOT_FOUND is remapped here, so any other failed status passes through as-is. The writeEntitiesIfConditionsHold javadoc says every failed condition or write gives TARGET_ENTITY_CONCURRENTLY_MODIFIED, and the transactional impl always returns that. Can we map every non-success status here, or loosen the javadoc?

🤖 AI-generated review comment — agents make mistakes, so please verify before acting on it.

* @param batch the conditions to check and the writes to apply
* @return the written entities in the order of the batch's writes, or a failure
*/
@NonNull ConditionalEntityWriteResult writeEntitiesIfConditionsHold(

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

New abstract SPI method (plus BasePersistence.checkAndWriteEntities), so out-of-tree backends stop compiling. AGENTS.md asks for that to go through the dev list. Is there a thread we can link, especially since #5820 fixes the same bug without changing the SPI?

🤖 AI-generated review comment — agents make mistakes, so please verify before acting on it.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@qlong
qlong marked this pull request as draft November 3, 2026 18:57
@qlong
qlong force-pushed the fix-5807-requirement-only-table-changes branch from 4b351df to f9d7f1c Compare November 3, 2026 19:32
@qlong
qlong marked this pull request as ready for review November 9, 2026 16:37

@kotwal-itpro kotwal-itpro left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this, @qlong. I read through the SPI, the JDBC and NoSQL implementations and the handler change, but haven't run it locally.

The handler side looks right to me. Comparing the resolved entity's metadata location with the metadata the requirements were validated against closes the window between loadTable and the resolve, and the atomic Unchanged check covers everything after that. testRequirementOnlyTableCommittedBetweenItsLoadAndItsResolveFailsTheCommit pins the first part.

One question on the JDBC side. checkEntityUnchanged takes SELECT ... FOR UPDATE on a row it only checks. The result stays correct, but two commits that only check the same table now wait for each other. They don't fail, as testBatchesThatOnlyCheckTheSameEntityDoNotConflict shows, but they run one at a time. When many commits assert the same table, which is the case Yufei raised on the list, they would queue on that one row.

FOR SHARE would let commits that only check the row run in parallel, and still make a concurrent update of the row wait until the batch commits. The (catalog_id, id) lock order would still rule out deadlocks, since a batch either checks or writes an entity, never both. PostgreSQL supports FOR SHARE. I'm not sure about H2, and I think CockroachDB only takes shared locks under READ COMMITTED by default, so this may need to depend on the database type. Is FOR UPDATE deliberate here, or would FOR SHARE work?

For context, the multi-table promotion doc in #5740 relies on the multi-table commit being atomic, so I'm glad to see #5807 fixed.

@qlong
qlong force-pushed the fix-5807-requirement-only-table-changes branch from f9d7f1c to fc23f9a Compare November 12, 2026 16:18
@kotwal-itpro

Copy link
Copy Markdown

Thanks @qlong, FOR SHARE on PostgreSQL with FOR UPDATE on H2 and CockroachDB looks good to me, and testGenerateCheckEntityQuery_lockClausePerDatabase pins it.

@iprithv iprithv left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this makes sense, checkEntityUnchanged goes through executeSelectOverStream(connection, …), and that overload still runs withRetries on the same connection.

We’re already inside runWithinTransaction here. If the SELECT hits 40001 (easy to see on Cockroach with max-retries>1, the docs example uses 3), the transaction is aborted, the retry gets something like 25P02 instead, and the 40001/40P01 mapping below never sees the original conflict. Caller ends up with a 500 instead of a concurrency failure / 409.

persistEntity’s execute(connection) path doesn’t retry, which is what we want inside a txn. Can we do the same for this select, skip the inner retry when we already own the connection?


/** {@inheritDoc} */
@Override
public @NonNull EntitiesResult updateEntitiesPropertiesIfNotChanged(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we deprecate it instead of replacing it with a new method?

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

will do

@ZephyrYWZhou

Copy link
Copy Markdown
Contributor

Thanks @qlong this reads well! Reducing the multi-table commit race to one general checkAndWriteEntities is nice. Two inline notes on the sealed types and on duplicates. +1 to @iprithv's point about the retry inside the transaction. FYI the DynamoDB backend gets a stub here; I think TransactWriteItems with a ConditionCheck can implement Unchanged later.

Set<EntityKey> entities = new HashSet<>();
for (Write write : writes) {
if (!(write instanceof Write.Update update)) {
throw new IllegalArgumentException("Unsupported write " + write);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since Write and Condition are sealed, could we use a switch over the sealed type in these places (here, AtomicOperationMetaStoreManager, JDBC, NoSQL, transactional)? Adding a new kind would then fail to compile in every backend, rather than throw IllegalArgumentException at run time. WDYT?

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point on error out at build time. polaris-core is jdk-17 so we cannot do exhaustive switch here. I will change jdbc / nosql backend to use switch for now

if (!(write instanceof Write.Update update)) {
throw new IllegalArgumentException("Unsupported write " + write);
}
requireOnce(entities, update.entity());

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If a multi-table commit lists the same table twice, once with updates and once requirement-only, this throws IllegalArgumentException. Does the REST layer reject duplicate identifiers before this point? If not, I think the client sees a 500 instead of a 4xx. Could we add a test for it?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed that it should be a 400 bad request in that case.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

commitTransaction group changes by table Id, so this probably won't happen but I think it is a good defensive coding to have. Will change to throw 400 if a condition write cannot be built

@qlong

qlong commented Nov 16, 2026 •

Copy link
Copy Markdown
Author

I think this makes sense, checkEntityUnchanged goes through executeSelectOverStream(connection, …), and that overload still runs withRetries on the same connection.

We’re already inside runWithinTransaction here. If the SELECT hits 40001 (easy to see on Cockroach with max-retries>1, the docs example uses 3), the transaction is aborted, the retry gets something like 25P02 instead, and the 40001/40P01 mapping below never sees the original conflict. Caller ends up with a 500 instead of a concurrency failure / 409.

persistEntity’s execute(connection) path doesn’t retry, which is what we want inside a txn. Can we do the same for this select, skip the inner retry when we already own the connection?

+1. Will remove retry within transaction.

I left DatasourceOperations.executeSelectOverStream(connection, …) as it is to not extend the scope of this PR, will fix it in follow up.

@qlong
qlong force-pushed the fix-5807-requirement-only-table-changes branch from fc23f9a to 8f10c69 Compare November 17, 2026 00:09
@adutra

adutra commented Nov 17, 2026

Copy link
Copy Markdown
Contributor

Heads-up: #5967 has been merged, so CHANGELOG.md is no longer edited directly. Changelog entries now go in a fragment file under changelog.d/, which avoids merge conflicts on CHANGELOG.md. See changelog.d/README.md for the format.

Could you rebase onto main, revert your changes to CHANGELOG.md, and add a fragment file instead if your change needs a changelog entry? A fragment is only needed for things worth calling out in release notes, such as breaking changes, metastore schema changes, or changes that require manual steps during an upgrade. Thanks!

Add conditional entity writes to the metastore SPI: a set of entity
writes applied atomically, and only if conditions on other entities
hold. Until now the only atomic condition a write could express was "the
entity being written is still at its version", so a decision based on
reading other entities could not be enforced at write time. This adds
the general primitive; this change uses it to fix apache#5807, and later
condition and write kinds (for example parent checks, create, drop) can
be added without changing the SPI.

Persistence:
- Add
  `PolarisMetaStoreManager.writeEntitiesIfConditionsHold(ConditionalEntityWrite)`.
  `Condition.Unchanged` checks an entity's version without writing it;
  `Write.Update` is the existing compare-and-swap update. If any
  condition or write fails, nothing is written and the result is
  `TARGET_ENTITY_CONCURRENTLY_MODIFIED`.
- Backends apply it through the new
  `BasePersistence.checkAndWriteEntities(conditions, changes)`.
  Conditions, writes and entity changes are sealed types at both layers,
  so later kinds (such as create and drop) are added without new
  methods.
- `updateEntitiesPropertiesIfNotChanged` is now a default method that
  delegates to it.
- Implement it on all backends. JDBC checks a row with
  `SELECT ... FOR SHARE` on PostgreSQL (`FOR UPDATE` on H2 and
  CockroachDB, which lack shared row locks by default) in the same
  transaction as the updates, takes row locks in (catalog_id, id) order,
  and reports deadlocks (40P01) and serialization failures (40001) as
  conflicts. NoSQL adds a `CHECK` mutation, evaluated in the same commit
  and again on retry. The DynamoDB backend rejects it, as it does its
  other not-yet-implemented multi-entity writes.

Fix (apache#5807):
- A multi-table commit can list a table with requirements but no
  updates, to assert it is unchanged. Such tables were validated when
  the commit was staged but were not part of the atomic write, so two
  concurrent commits that each changed one table and asserted the other
  could both succeed.
- `commitTransaction` now adds each requirement-only table as an
  `Unchanged` condition, after verifying that its entity still points to
  the metadata the requirements were validated against. Commits that
  only assert the same table do not fail each other.

Tests: conformance tests on all four backends (including PostgreSQL),
with concurrent writers for write skew and crossed updates; a NoSQL test
that forces a retry after a failed compare-and-swap; handler tests that
inject a concurrent commit at the two points of the race.

Fixes apache#5807
- JDBC: run the check's select once inside the transaction, so that a
  serialization failure reaches the conflict mapping instead of being
  retried within the aborted transaction.
- Deprecate updateEntitiesPropertiesIfNotChanged in favour of
  writeEntitiesIfConditionsHold.
- JDBC and NoSQL: exhaustive switch over the sealed condition and write
  kinds, so a new kind fails to compile there.
- commitTransaction: reject a conditional write that cannot be built
  with a 400, and clean up the metadata files already written when the
  commit fails before the conditional write is applied.
CHANGELOG.md is no longer edited directly since apache#5967. Revert the
entries added to it and add them as a fragment instead, with a new
Deprecations entry for updateEntitiesPropertiesIfNotChanged.
@qlong
qlong force-pushed the fix-5807-requirement-only-table-changes branch from 8f10c69 to 88fa024 Compare November 19, 2026 01:40
@qlong

qlong commented Nov 19, 2026

Copy link
Copy Markdown
Author

Heads-up: #5967 has been merged, so CHANGELOG.md is no longer edited directly. Changelog entries now go in a fragment file under changelog.d/, which avoids merge conflicts on CHANGELOG.md. See changelog.d/README.md for the format.

Could you rebase onto main, revert your changes to CHANGELOG.md, and add a fragment file instead if your change needs a changelog entry? A fragment is only needed for things worth calling out in release notes, such as breaking changes, metastore schema changes, or changes that require manual steps during an upgrade. Thanks!

Added a fragment for SPI change.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Multi-table commit does not isolate requirements on tables without updates: concurrent transactions both succeed

7 participants