[default values] Gate column-default rewrites on every branch, not only main

[cbb330]

Summary

Fixes #693. For a table with column defaults, Type 2 requires an overwrite or replace commit to send a matching initial-default, so only default-aware clients can rewrite it. The check looked only at the snapshot main points to. An unaware client could still rewrite the table on a named branch or as a staged WAP snapshot, and creating a branch or tag at an existing overwrite was rejected as a rewrite.

The snapshots PUT already carries the table's full snapshot list. This change gates the overwrite and replace snapshots in that list that the table has not persisted yet, whichever ref they are on. It uses the existing request fields, so there is no API change. It compares snapshot ids, as the commit path already does to find new snapshots, and does not diff ref maps, which #693 rules out.

On #681, below the client gate (#774), which merges last. The code does not depend on the PRs below it; the stack keeps the column-default changes in one merge order.

Changes

• Client-facing API Changes

• Internal API Changes

• Bug Fixes

• New Features

• Performance Improvements

• Code Style

• Refactoring

• Documentation

• Tests

• Rewrite detection: ReadBridgeStripProtection treats a commit as a rewrite when it is a replace, or when jsonSnapshots contains an overwrite or replace snapshot the table has not persisted. Snapshots already on the table are history, so ref-only commits (create a branch or tag, roll back to an existing snapshot) are not gated.

• Persisted snapshot ids: the new OpenHouseInternalRepository.findSnapshotIds, wired in ApiConfig, returns every snapshot id in the table's current metadata, including staged snapshots no ref points to. It is read only for a stamped, ramped table whose PUT lists an overwrite or replace snapshot, including older ones still in history. Those commits cost one extra metadata load; tables without column defaults pay nothing.

• Failure: if the lookup fails, the commit fails closed with the lookup's own exception. The request is not at fault, so it is not reported as a 400; the exception advice handles it like any other catalog failure.

Commit on a table with column defaults	Before	After
Overwrite on a named branch	accepted	needs a matching initial-default
Staged WAP overwrite (no ref yet)	accepted	needs a matching initial-default
Overwrite on main	needs a matching initial-default	unchanged
Branch or tag created at an existing overwrite	rejected	accepted
Append with an older overwrite in history	accepted	unchanged

Testing Done

• Manually Tested on local docker setup. Please include commands ran, and their output.

• Added new tests for the changes made.

• Updated existing tests to reflect the changes made.

• No tests added or updated. Please explain why. If unsure, please feel free to ask for help.

• Some other form of testing like staging or soak time in production. Please explain.

• ReadBridgeStripProtectionTest: a branch overwrite next to a main append and a staged WAP overwrite are rejected; a branch and tag at a persisted overwrite, and a historical overwrite under a current append, are accepted; a failing lookup fails closed with its own exception.

• ReadBridgeColumnDefaultE2ETest.snapshotsPut_gatesAddedBranchRewritesButNotRefsAtPersistedRewrites drives the snapshots endpoint through the real service, repository and catalog. A branch overwrite returns 400 COLUMN_DEFAULT_REWRITE, a handshake overwrite on main commits, and a branch plus tag at that overwrite commit. Without the fix, on main at 5ee734d2, the branch overwrite returned 200.

• On this stack, with JDK 11: :services:tables:test for readbridge.*, ReadBridgeColumnDefaultE2ETest, SnapshotsControllerTest, mock.*, services.*, repository.* and config.* (686 tests) and :client:secureclient:test (20 tests) pass.

• Rebuilt below the client gate: the end-to-end test creates its table and PUTs snapshots without the gate's Spark-client headers. :services:tables:test (923 tests) passes and spotlessCheck is clean.

• Restacked on [default values] Fail loudly when column defaults cannot be applied #790's typed failures (6ef9c8a9): a failing lookup no longer becomes COLUMN_DEFAULT_UNUSABLE (400). ReadBridgeStripProtectionTest passes on this commit; at the top of the stack, :services:tables:test (924 tests) passes and spotlessCheck is clean.

Additional Information

• Breaking Changes
• Deprecations
• Large PR broken into smaller PRs, and PR plan linked in the description.

Stack

Stack #798, bottom to top:

1. #793: report the client release in the User-Agent
2. #790: fail loudly when column defaults cannot be applied
3. #797: make the opt-in immutable once set to true, with the policy override
4. #681: Spark catalog itest (overlay)
5. This PR: gate column-default rewrites on every branch
6. #774: gate opted-in table IO on Spark client version (merges last)