Skip to content

fix(core): large response body logging causes memory growth#13581

Draft
AlinsRan wants to merge 1 commit into
apache:masterfrom
AlinsRan:fix/11244-large-body-memory
Draft

fix(core): large response body logging causes memory growth#13581
AlinsRan wants to merge 1 commit into
apache:masterfrom
AlinsRan:fix/11244-large-body-memory

Conversation

@AlinsRan

Copy link
Copy Markdown
Contributor

Description

Fixes #11244

Root cause

core.response.hold_body_chunk accumulated every response-body chunk into a Lua table ({chunk, n, bytes, ...}) and, when the size limit was reached or on eof, concatenated the whole table into one string with table.concat. For large bodies (the issue reports ~128K), this keeps both the list of individual chunks and the final concatenated string alive simultaneously, and each table.concat allocates a brand new string. The result is excessive allocation/retention that shows up as memory growth when loggers hold large response bodies.

Fix

Rewrite the accumulation to use a LuaJIT string.buffer (require("string.buffer")), reviving the approach from the abandoned #12035 and adapting it to the current hold_body_chunk shape:

  • chunks are appended with buffer:put(chunk) instead of being stored in a growing Lua table;
  • the final body is produced with buffer:get() (full body) or buffer:get(max_resp_body_bytes) (truncated), which returns exactly the first max_resp_body_bytes bytes and avoids materializing an intermediate concatenated copy before truncation;
  • the wrapper table is kept only for the bytes counter and the done flag so the buffer object can carry per-buffer_key state.

All existing semantics are preserved:

  • the hold_the_copy flag (still flushes arg[1] on non-final chunks when false);
  • max_resp_body_bytes truncation (returns exactly max_resp_body_bytes);
  • the done flag (later chunks and eof return nil after a truncated body was already returned);
  • the single-chunk-that-is-also-eof path;
  • per-body_buffer_key / ctx._plugin_name storage.

Test

Adds t/core/response-hold-body.t covering:

  1. multi-chunk accumulation of a 128K body (full body returned at eof);
  2. truncation at max_resp_body_bytes across multiple chunks;
  3. the done flag — later chunks and eof return nil after truncation;
  4. a single chunk that is also eof, truncated to max_resp_body_bytes;
  5. small-body single-chunk passthrough (mirrors the existing t/core/response.t TEST 8 mock pattern).

Verification

  • luacheck apisix/core/response.lua passes (0 warnings / 0 errors).
  • All inline Lua blocks in the new .t parse cleanly.
  • The rewritten function was exercised end-to-end via a standalone harness that drives the same ngx.arg[1]/[2] chunk/eof protocol; all six scenarios above pass and return byte-identical bodies (e.g. truncation returns exactly 131072 bytes).
  • A memory micro-benchmark inside test-nginx is not reliable, so the rationale is documented above; the string.buffer path avoids the simultaneous chunk-table + concatenated-string retention that caused the growth.

Note: the full prove test-nginx run was not executed in my local environment because the available OpenResty build (1.21.4.4) does not support the quic/http3 listen directives that t/APISIX.pm emits unconditionally on master; this is an environment limitation, not a test failure. CI runs against a QUIC-capable OpenResty.

Checklist

  • I have explained the need for this PR and the problem it solves
  • I have explained the changes or the new features added to this PR
  • I have added tests corresponding to this change
  • I have updated the documentation to reflect this change
  • I have verified that this change is backward compatible

hold_body_chunk accumulated every response body chunk into a Lua table
and, once the size limit was reached or on eof, concatenated the whole
table into a single string with table.concat. For large bodies (the
issue reports ~128K), this keeps both the list of chunks and the final
concatenated string alive at the same time, and every concat allocates
a fresh string, causing excessive memory allocation/retention.

Rewrite the accumulation to use a LuaJIT string.buffer
(require("string.buffer")): chunks are appended with buffer:put and the
final body is produced with buffer:get, which avoids holding the chunk
table plus an intermediate concatenated copy. For the truncation path
buffer:get(max_resp_body_bytes) returns exactly the first
max_resp_body_bytes bytes; the buffer is then dropped once `done` is set.

All existing semantics are preserved: the hold_the_copy flag,
max_resp_body_bytes truncation, the done flag (later chunks/eof return
nil after truncation), the single-chunk eof path and the per-buffer-key
storage.

Add t/core/response-hold-body.t covering multi-chunk accumulation, byte
truncation across chunks, the done flag, the single-chunk eof case and
small-body passthrough.

Fixes apache#11244
@AlinsRan AlinsRan force-pushed the fix/11244-large-body-memory branch from 029861f to b4cb6cb Compare June 22, 2026 03:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bug: Memory leak when logger log the traffic with large (about 128K) response body

1 participant