Skip to content

feat(s3stream): recycle sequential byte buffer slabs#3463

Merged
superhx merged 1 commit into
mainfrom
feat/recycling-byte-buf-seq-alloc-main
Jul 17, 2026
Merged

feat(s3stream): recycle sequential byte buffer slabs#3463
superhx merged 1 commit into
mainfrom
feat/recycling-byte-buf-seq-alloc-main

Conversation

@superhx

@superhx superhx commented Jul 17, 2026

Copy link
Copy Markdown
Collaborator

Summary

Introduce RecyclingByteBufSeqAlloc to retain and reuse sequential allocation
slabs instead of repeatedly allocating and releasing large backing buffers.

This change also makes record encode/decode allocator ownership explicit and
migrates applicable ByteBufSeqAlloc users to the recycling allocator.

Motivation

With POOLED_HEAP, Netty's 4 MiB heap chunk is backed by a byte[]. After the
array header and object alignment are included, the object exceeds 4 MiB and
occupies 6 MiB of ZGC large-page capacity.

The previous sequential allocator also released its backing buffer after all
retained slices were released. Under sustained append and batched LogCache
eviction, this caused large buffers to repeatedly enter old generation and wait
for collection instead of being reused.

Implementation

  • Add RecyclingByteBufSeqAlloc with:
    • sequential retained-slice allocation;
    • retired slab detection through owner refCnt;
    • LIFO reuse of free slabs;
    • one-minute idle eviction;
    • periodic maintenance for low-traffic workloads;
    • heap and direct allocation support;
    • thread-safe allocation and close behavior.
  • Set heap slab payload to 4 MiB - 64 B, keeping the backing array within
    4 MiB of ZGC page capacity.
  • Source direct slabs from ByteBufAlloc so they continue using Netty's pooled
    allocator.
  • Scale allocator concurrency with:
    min(CPU, max(heap size / 2 GiB, 1)).
  • Export retained slab memory through
    kafka_stream_buffer_allocated_memory_size with source="seq_alloc".
  • Add the source label to existing ByteBufAlloc metric series.
  • Replace applicable sequential allocators in stream append, WAL, snapshot,
    compaction, link-record, and block-cache paths.
  • Require explicit allocators for StreamRecordBatch encode/decode operations.
  • Distinguish copied and retained ObjectReader.DataBlockGroup iteration.
  • Classify copied block-cache records as BLOCK_CACHE.
  • Add allocator unit coverage and include affected AutoMQ tests in S3Unit.

Process-wide allocators intentionally remain alive for the process lifetime so
unrelated instances of the same component can reuse a bounded active slab set.

Compatibility

  • No configuration changes are required.
  • The allocator mode follows the process-wide ByteBufAlloc policy selected
    during startup.
  • Existing allocated-memory metrics gain a source="byte_buf_alloc" label.
    Recycling allocator series use source="seq_alloc".

## Summary

Introduce `RecyclingByteBufSeqAlloc` to retain and reuse sequential
allocation
slabs instead of repeatedly allocating and releasing large backing
buffers.

This change also makes record encode/decode allocator ownership explicit
and
migrates applicable `ByteBufSeqAlloc` users to the recycling allocator.

## Motivation

With `POOLED_HEAP`, Netty's 4 MiB heap chunk is backed by a `byte[]`.
After the
array header and object alignment are included, the object exceeds 4 MiB
and
occupies 6 MiB of ZGC large-page capacity.

The previous sequential allocator also released its backing buffer after
all
retained slices were released. Under sustained append and batched
LogCache
eviction, this caused large buffers to repeatedly enter old generation
and wait
for collection instead of being reused.

## Implementation

- Add **`RecyclingByteBufSeqAlloc`** with:
  - sequential retained-slice allocation;
  - retired slab detection through owner `refCnt`;
  - LIFO reuse of free slabs;
  - one-minute idle eviction;
  - periodic maintenance for low-traffic workloads;
  - heap and direct allocation support;
  - thread-safe allocation and close behavior.
- Set heap slab payload to `4 MiB - 64 B`, keeping the backing array
within
  4 MiB of ZGC page capacity.
- Source direct slabs from `ByteBufAlloc` so they continue using Netty's
pooled
  allocator.
- Scale allocator concurrency with:
  `min(CPU, max(heap size / 2 GiB, 1))`.
- Export retained slab memory through
  `kafka_stream_buffer_allocated_memory_size` with `source="seq_alloc"`.
- Add the `source` label to existing `ByteBufAlloc` metric series.
- Replace applicable sequential allocators in stream append, WAL,
snapshot,
  compaction, link-record, and block-cache paths.
- Require explicit allocators for `StreamRecordBatch` encode/decode
operations.
- Distinguish copied and retained `ObjectReader.DataBlockGroup`
iteration.
- Classify copied block-cache records as `BLOCK_CACHE`.
- Add allocator unit coverage and include affected AutoMQ tests in
`S3Unit`.

Process-wide allocators intentionally remain alive for the process
lifetime so
unrelated instances of the same component can reuse a bounded active
slab set.

## Compatibility

- No configuration changes are required.
- The allocator mode follows the process-wide `ByteBufAlloc` policy
selected
  during startup.
- Existing allocated-memory metrics gain a `source="byte_buf_alloc"`
label.
  Recycling allocator series use `source="seq_alloc"`.
@superhx

superhx commented Jul 17, 2026

Copy link
Copy Markdown
Collaborator Author

This PR cherry-picks #3461 into the main branch.

@superhx
superhx enabled auto-merge (squash) July 17, 2026 06:29
@superhx
superhx merged commit 9de53c7 into main Jul 17, 2026
7 checks passed
@superhx
superhx deleted the feat/recycling-byte-buf-seq-alloc-main branch July 17, 2026 06:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants