Skip to content

fix(inkless:consolidation): DeleteRecords across tiers on consolidating topics#707

Draft
viktorsomogyi wants to merge 1 commit into
svv/ts-unification-docsfrom
svv/ts-unification-delete-records
Draft

fix(inkless:consolidation): DeleteRecords across tiers on consolidating topics#707
viktorsomogyi wants to merge 1 commit into
svv/ts-unification-docsfrom
svv/ts-unification-delete-records

Conversation

@viktorsomogyi

Copy link
Copy Markdown
Contributor

Makes DeleteRecords correct end-to-end for switched consolidating diskless topics (both diskless.enable and remote.storage.enable), where the client-facing leader is a hash/AZ-selected replica (via the metadata transformer) rather than the real KRaft leader, and a freshly-rebuilt leader's local log start is pinned at the classic-to-diskless seal.

Changes, in layers:

  • Routing (local leg → real leader). DisklessDeleteRecordsForwarder forwards the local leg of a DeleteRecords request to the actual leader, so the admin request no longer loops on NOT_LEADER_OR_FOLLOWER against a transformer-advertised follower. Wired through KafkaApis/KafkaApisBuilder/BrokerServer. DeleteRecordsCommand now exits non-zero on per-partition errors.
  • Cross-tier low watermark. For consolidating partitions the completion low watermark and ListOffsets(EARLIEST) come from the authoritative control-plane cross-tier earliest, not the ISR (whose follower log starts are frozen at the switch). DisklessFetchOffsetRouter routes EARLIEST to the control plane (Problem A: cross-broker earliest consistency), and DisklessLeaderEndPoint reports min(crossTierEarliest, localLogStart) as the whole-log start.
  • Reclaim floor (Problem B). RemoteLogManager takes a logStartOffsetOverride (control-plane cross-tier earliest) so a rebuilt leader no longer reclaims the entire remote classic prefix [earliest, seal).
  • Fail-safe reclaim floor. When the override is unavailable (control plane down / metadata not yet propagated), the RLM no longer falls back to the local seal for a consolidating partition — it uses the remote earliest and defers, so a transient control-plane outage can never trigger irreversible over-deletion. A new isConsolidatingDisklessPartition predicate distinguishes this from classic topics, which keep the upstream local-log-start floor.

Docs: DISKLESS_CONSOLIDATION.md updated; full design/rationale and an uncertainties/risk-tracking checklist in DELETE_RECORDS_DISKLESS_ROUTING_IMPL.md.

  • RemoteLogManagerTest — reclaim floor uses the override; fail-safe skips reclaim when the override is absent for consolidating; classic topics unchanged.
  • ReplicaManagerInklessTestcrossTierEarliestOffset branches (cache hit, miss→CP, CP error/throw→empty, non-consolidating→empty); isConsolidatingDisklessPartition.
  • DisklessLeaderEndPointTest — whole-log start = cross-tier earliest; reverts to local seal when unavailable.
  • DisklessFetchOffsetRouterTest, DisklessDeleteRecordsForwarderTest, DeleteRecordsCommandTest.
  • e2e consolidation_delete_records_across_tiers_test.py — earliest advances off 0, is identical on every broker, settles exactly at the delete boundary, and the surviving prefix reads back contiguous.
  • core/storage compile + checkstyle + spotbugs green.
  • Open items tracked in the checklist in DELETE_RECORDS_DISKLESS_ROUTING_IMPL.md (multi-partition/rapid-failover e2e, retention×DeleteRecords, sync-DB/batching, etc.).

Delete this text and replace it with a detailed description of your change. The
PR title and body will become the squashed commit message.

If you would like to tag individuals, add some commentary, upload images, or
include other supplemental information that should not be part of the eventual
commit message, please use a separate comment.

If applicable, please include a summary of the testing strategy (including
rationale) for the proposed change. Unit and/or integration tests are expected
for any behavior change and system tests should be considered for larger
changes.

…ng topics

Makes `DeleteRecords` correct end-to-end for **switched consolidating** diskless
topics (both `diskless.enable` and `remote.storage.enable`), where the client-facing
leader is a hash/AZ-selected replica (via the metadata transformer) rather than the
real KRaft leader, and a freshly-rebuilt leader's local log start is pinned at the
classic-to-diskless seal.

Changes, in layers:

- **Routing (local leg → real leader).** `DisklessDeleteRecordsForwarder` forwards the
  local leg of a `DeleteRecords` request to the actual leader, so the admin request no
  longer loops on `NOT_LEADER_OR_FOLLOWER` against a transformer-advertised follower.
  Wired through `KafkaApis`/`KafkaApisBuilder`/`BrokerServer`. `DeleteRecordsCommand`
  now exits non-zero on per-partition errors.
- **Cross-tier low watermark.** For consolidating partitions the completion low
  watermark and `ListOffsets(EARLIEST)` come from the authoritative control-plane
  cross-tier earliest, not the ISR (whose follower log starts are frozen at the switch).
  `DisklessFetchOffsetRouter` routes `EARLIEST` to the control plane (**Problem A**:
  cross-broker earliest consistency), and `DisklessLeaderEndPoint` reports
  `min(crossTierEarliest, localLogStart)` as the whole-log start.
- **Reclaim floor (Problem B).** `RemoteLogManager` takes a `logStartOffsetOverride`
  (control-plane cross-tier earliest) so a rebuilt leader no longer reclaims the entire
  remote classic prefix `[earliest, seal)`.
- **Fail-safe reclaim floor.** When the override is unavailable (control plane down /
  metadata not yet propagated), the RLM no longer falls back to the local seal for a
  *consolidating* partition — it uses the remote earliest and defers, so a transient
  control-plane outage can never trigger irreversible over-deletion. A new
  `isConsolidatingDisklessPartition` predicate distinguishes this from classic topics,
  which keep the upstream local-log-start floor.

Docs: `DISKLESS_CONSOLIDATION.md` updated; full design/rationale and an
uncertainties/risk-tracking checklist in `DELETE_RECORDS_DISKLESS_ROUTING_IMPL.md`.

- [x] `RemoteLogManagerTest` — reclaim floor uses the override; fail-safe skips reclaim
      when the override is absent for consolidating; classic topics unchanged.
- [x] `ReplicaManagerInklessTest` — `crossTierEarliestOffset` branches (cache hit,
      miss→CP, CP error/throw→empty, non-consolidating→empty); `isConsolidatingDisklessPartition`.
- [x] `DisklessLeaderEndPointTest` — whole-log start = cross-tier earliest; reverts to
      local seal when unavailable.
- [x] `DisklessFetchOffsetRouterTest`, `DisklessDeleteRecordsForwarderTest`, `DeleteRecordsCommandTest`.
- [x] e2e `consolidation_delete_records_across_tiers_test.py` — earliest advances off 0,
      is identical on every broker, settles exactly at the delete boundary, and the
      surviving prefix reads back contiguous.
- [x] `core`/`storage` compile + checkstyle + spotbugs green.
- [ ] Open items tracked in the checklist in `DELETE_RECORDS_DISKLESS_ROUTING_IMPL.md`
      (multi-partition/rapid-failover e2e, retention×DeleteRecords, sync-DB/batching, etc.).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant