fix(fts): use async send in FTS index builder to prevent thread-pool …#7423
Open
a-agmon wants to merge 1 commit into
Open
fix(fts): use async send in FTS index builder to prevent thread-pool …#7423a-agmon wants to merge 1 commit into
a-agmon wants to merge 1 commit into
Conversation
Author
|
Hi @westonpace - would be happy for your review. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes lancedb/lancedb#3568 (the issue arises in lancedb indexing)
Building a full-text-search index hangs permanently at 0% CPU on hosts
whose Lance CPU pool has a single thread.
The CPU compute pool is sized
max(1, num_cpus - LANCE_IO_CORE_RESERVATION)(default reservation 2), so any machine with <= 3 visible CPUs (1-vCPU VMs, CI runners, CPU-limited Kubernetes pods) collapses to a 1-thread pool and deadlocks.Root cause is in
write_posting_lists. The posting-list producer runs on the CPU pool viaspawn_cpuand pushes batches into a capacity-1async_channelusing the synchronoustx.send_blocking(). When the channel is full,send_blockingparks the OS thread it is running on. On a single-thread pool that is the only thread, and the async consumer's column encoder (write_record_batch->spawn_cpu) needs that same pool to drain the channel. The parked producer and the starved consumer wait on each other forever: no timeout, no error, just a silent hang at 0% CPU.The hang only triggers once the posting lists span a second output batch (so the producer reaches a second, blocking send), which is why it appears as a data-size "cliff".
The PR restructures the producer as an async task that builds each batch on the CPU pool via
spawn_cpuand dispatches it withtx.send(batch).await. When the channel is full,send().awaityields the task back to the runtime instead of parking a pool thread, so the consumer can always be scheduled to drain it. Between batches the producer holds no pool thread while waiting, making the pool size irrelevant. The builder and the remaining posting-list iterator are handed back out of eachspawn_cpucall so the cross-batch cache-group accumulator is preserved.In addition, it adds a regression test that writes a partition whose posting lists span many output batches (exercising channel back-pressure) under a timeout and verifies every batch is searchable.
(verbose comments added in the code intentionally for review purposes - can be removed if inappropriate. I just thought it might be helpful as the issue is somewhat confusing)