feat(s3stream): recycle sequential byte buffer slabs#3463
Merged
Conversation
## Summary Introduce `RecyclingByteBufSeqAlloc` to retain and reuse sequential allocation slabs instead of repeatedly allocating and releasing large backing buffers. This change also makes record encode/decode allocator ownership explicit and migrates applicable `ByteBufSeqAlloc` users to the recycling allocator. ## Motivation With `POOLED_HEAP`, Netty's 4 MiB heap chunk is backed by a `byte[]`. After the array header and object alignment are included, the object exceeds 4 MiB and occupies 6 MiB of ZGC large-page capacity. The previous sequential allocator also released its backing buffer after all retained slices were released. Under sustained append and batched LogCache eviction, this caused large buffers to repeatedly enter old generation and wait for collection instead of being reused. ## Implementation - Add **`RecyclingByteBufSeqAlloc`** with: - sequential retained-slice allocation; - retired slab detection through owner `refCnt`; - LIFO reuse of free slabs; - one-minute idle eviction; - periodic maintenance for low-traffic workloads; - heap and direct allocation support; - thread-safe allocation and close behavior. - Set heap slab payload to `4 MiB - 64 B`, keeping the backing array within 4 MiB of ZGC page capacity. - Source direct slabs from `ByteBufAlloc` so they continue using Netty's pooled allocator. - Scale allocator concurrency with: `min(CPU, max(heap size / 2 GiB, 1))`. - Export retained slab memory through `kafka_stream_buffer_allocated_memory_size` with `source="seq_alloc"`. - Add the `source` label to existing `ByteBufAlloc` metric series. - Replace applicable sequential allocators in stream append, WAL, snapshot, compaction, link-record, and block-cache paths. - Require explicit allocators for `StreamRecordBatch` encode/decode operations. - Distinguish copied and retained `ObjectReader.DataBlockGroup` iteration. - Classify copied block-cache records as `BLOCK_CACHE`. - Add allocator unit coverage and include affected AutoMQ tests in `S3Unit`. Process-wide allocators intentionally remain alive for the process lifetime so unrelated instances of the same component can reuse a bounded active slab set. ## Compatibility - No configuration changes are required. - The allocator mode follows the process-wide `ByteBufAlloc` policy selected during startup. - Existing allocated-memory metrics gain a `source="byte_buf_alloc"` label. Recycling allocator series use `source="seq_alloc"`.
Collaborator
Author
|
This PR cherry-picks #3461 into the |
superhx
enabled auto-merge (squash)
July 17, 2026 06:29
Gezi-lzq
approved these changes
Jul 17, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Introduce
RecyclingByteBufSeqAllocto retain and reuse sequential allocationslabs instead of repeatedly allocating and releasing large backing buffers.
This change also makes record encode/decode allocator ownership explicit and
migrates applicable
ByteBufSeqAllocusers to the recycling allocator.Motivation
With
POOLED_HEAP, Netty's 4 MiB heap chunk is backed by abyte[]. After thearray header and object alignment are included, the object exceeds 4 MiB and
occupies 6 MiB of ZGC large-page capacity.
The previous sequential allocator also released its backing buffer after all
retained slices were released. Under sustained append and batched LogCache
eviction, this caused large buffers to repeatedly enter old generation and wait
for collection instead of being reused.
Implementation
RecyclingByteBufSeqAllocwith:refCnt;4 MiB - 64 B, keeping the backing array within4 MiB of ZGC page capacity.
ByteBufAllocso they continue using Netty's pooledallocator.
min(CPU, max(heap size / 2 GiB, 1)).kafka_stream_buffer_allocated_memory_sizewithsource="seq_alloc".sourcelabel to existingByteBufAllocmetric series.compaction, link-record, and block-cache paths.
StreamRecordBatchencode/decode operations.ObjectReader.DataBlockGroupiteration.BLOCK_CACHE.S3Unit.Process-wide allocators intentionally remain alive for the process lifetime so
unrelated instances of the same component can reuse a bounded active slab set.
Compatibility
ByteBufAllocpolicy selectedduring startup.
source="byte_buf_alloc"label.Recycling allocator series use
source="seq_alloc".