Skip to content

fix: memory leak in CompactionUploader during shutdown#3464

Open
vishalmaurya850 wants to merge 1 commit into
AutoMQ:mainfrom
vishalmaurya850:abortcompaction
Open

fix: memory leak in CompactionUploader during shutdown#3464
vishalmaurya850 wants to merge 1 commit into
AutoMQ:mainfrom
vishalmaurya850:abortcompaction

Conversation

@vishalmaurya850

Copy link
Copy Markdown

Fix memory leak in CompactionUploader during shutdown

More detailed description of your change, if necessary.

During shutdown, CompactionUploader leaked ByteBuf objects allocated by ProxyWriter.ObjectWriter because in-flight writeStreamObject operations were not aware that the uploader was shutting down.

Root cause: shutdown() never set isAborted = true. This allowed in-flight lambdas in writeStreamObject and prepareObjectAndWrite to pass the abort check, allocate a DataBlockWriter (which allocates a CompositeByteBuf internally), and then chain writes on a pool that was terminating. If pending StreamDataBlock.getDataCf() futures never completed (because upstream readers were also shutting down), the future chain hung indefinitely and the whenComplete cleanup handler was never invoked — leaking the writer's internal buffer.

Changes:

  • Set isAborted = true in shutdown() so in-flight operations bail out early.
  • Release streamSetObjectWriter after pools terminate in shutdown().
  • Add a double-check of isAborted in writeStreamObject() immediately after DataBlockWriter allocation to close the race window between the first check and writer creation.
  • Add an isAborted guard in prepareObjectAndWrite() before writing to the stream-set object writer.

Issue

Fixes #2364

Summary of testing strategy

  • Existing CompactionUploaderTest (unit tests for writeStreamObject, chainWriteStreamSetObject) continues to pass, confirming no regression in the happy path.
  • The fix is a defensive shutdown guard — the race condition is inherently timing-dependent and difficult to reproduce deterministically in a unit test. Verification relies on the absence of ResourceLeakDetector LEAK warnings in integration/E2E runs under repeated compaction-shutdown cycles.
  • Netty's ResourceLeakDetector (set to PARANOID in test environments) will surface any remaining leaks if the fix is incomplete.

Committer Checklist (excluded from commit message)

  • Verify design and implementation
  • Verify test coverage and CI build status
  • Verify documentation (including upgrade notes)

Signed-off-by: Vishal Maurya <vishalmaurya850@gmail.com>
@vishalmaurya850 vishalmaurya850 changed the title Add abort handling in shutdown process fix: memory leak in CompactionUploader during shutdown Jul 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Memory Leak in CompactionUploader During Shutting Down

1 participant