[harmadasg]

Dynamic thread count updates must be within [current / 2, current * 2]. On restart, the thread pools are created from the static configs, and the dynamic configs are applied by the first metadata publish. That publish folds all changes into a single update, so a sequence of valid steps such as num.io.threads 16 -> 8 -> 6 arrives as 16 -> 6 and is rejected. The node keeps the static thread counts.

KAFKA-17431 reads the dynamic broker configs from the latest snapshot on startup, but changes committed after that snapshot still go through the first publish.

My proposed fix skips the [current / 2, current * 2] bounds while the first publish applies the dynamic broker configs, on both brokers and controllers. The later updates are validated as before. Thread count configs in scope for the fix:

• num.io.threads
• num.network.threads
• num.replica.fetchers
• num.recovery.threads.per.data.dir
• background.threads
• log.cleaner.threads
• the remote log manager thread pools
• mirror.num.replica.fetchers

[harmadasg]

Hi @kevin-wu24, since you worked on the related issue KAFKA-17431, would you mind taking a look if you have some time? Thanks in advance

[harmadasg]

The recent failures in the newly added testDynamicThreadCountsAreAppliedAfterRestart uncovered an existing bug outside of my recent changes. In short, shutdown can hang if it starts right after a request handler thread-pool shrink event. (num.io.threads)

1. KafkaRequestHandlerPool.resizeThreadPool shrinks the pool by calling stop() on the removed handlers
2. A stopped handler only exits after its current receiveRequest(300) poll returns.
3. If shutdown() runs during that window, it sends one ShutdownRequest per remaining handler
4. A removed handler can take one of those requests and exit, so a remaining handler never gets one and awaitShutdown() blocks forever

This scenario is not likely to happen on a real broker as the full shutdown process takes multiple seconds and the shutdown has to start within ~300ms of a shrink. But still it can affect existing integration tests. I pushed e7b3f86 to fix the shutdown race. Also added a new unit test to verify the change. As mentioned the bug exists on trunk independently, the new test just makes it reachable so the new test should fail against trunk. The test is not 100% deterministic but shrinking the pool from 16 to 1 and then shutdown has a high chance that the shutdown request will find one of the 15 removed handlers.

While working on a fix I also noticed another problematic part where filtering for to be removed handlers can prevent a possible zero division exception