Repository navigation
Conversation
The S3 -> raw table copy during normalize issued one INSERT per batch file. ClickHouse decodes a single Avro file on one stream, so each copy was bound to one core regardless of max_threads/max_insert_threads. Add PEERDB_CLICKHOUSE_RAW_COPY_FILES_PER_INSERT (default 1) to load N batch files with a single INSERT ... SELECT ... UNION ALL ..., one branch per file, letting ClickHouse read and decode them in parallel. The raw table checkpoint advances per INSERT. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
dtunikov
marked this pull request as ready for review
October 30, 2026 12:46
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Normalize copies staged Avro files into the raw table one INSERT per batch. ClickHouse decodes a single Avro file on one stream, so this step is bound to ~1 core (seen in prod: ~55s CPU ≈ 57s wall per 100k-row batch, with
max_threads=89).max_insert_threadsdoes not help since the source is the bottleneck.Adds
PEERDB_CLICKHOUSE_RAW_COPY_FILES_PER_INSERT(default1, current behavior). With N > 1, up to N batch files are loaded by oneINSERT ... SELECT * FROM s3(f1) UNION ALL SELECT * FROM s3(f2) ..., so each file gets its own stream. Rows already carry_peerdb_batch_id, so no per-file metadata is needed. Works for both S3 and GCS (signedurl()) staging. The raw table checkpoint advances per INSERT.Default is kept at 1 because a single-file raw insert already peaks at ~15 GB memory for large documents; N files in parallel can multiply that.
🤖 Generated with Claude Code