Skip to content

[branch-4.1](zone-map) Avoid per-row Field temporaries in TypedZoneMapIndexWriter (#62544)#63050

Merged
yiguolei merged 1 commit into
apache:branch-4.1from
csun5285:tmp/cherry-pick-62544-to-4.1
May 9, 2026
Merged

[branch-4.1](zone-map) Avoid per-row Field temporaries in TypedZoneMapIndexWriter (#62544)#63050
yiguolei merged 1 commit into
apache:branch-4.1from
csun5285:tmp/cherry-pick-62544-to-4.1

Conversation

@csun5285
Copy link
Copy Markdown
Contributor

@csun5285 csun5285 commented May 7, 2026

pick from master #62544

…riter (apache#62544)

When writing nullable columns, `append_nullable` splits each block at
null boundaries into small non-null runs.

| Input block | add_values calls |

|--------------------------------------|-----------------------------------------------------------|
| `[1, 3, NULL, NULL, 4, 5, 6, NULL]` | `add_values([1,3], count=2)` +
`add_values([4,5,6], count=3)` |

Each `add_values` call built two heap-allocated `Field` temporaries just
to compare
min/max — allocation, memcpy, variant dispatch, destructor — per call.

Track page min/max as raw `ValType` (the same native type
`std::minmax_element` already uses). Materialize `Field` only once at
`flush()` time.

`block_rows` = post-null-split run length.

| Type | block | Baseline (M items/s) | Optimized (M items/s) | Speedup
|
|---|---:|---:|---:|---:|
| Int32 | 1 | 21.9 | 240.3 | **11.0×** |
| Int32 | 4 | 75.9 | 353.0 | **4.7×** |
| Int32 | 16 | 170.0 | 400.1 | **2.4×** |
| Int32 | 64 | 261.9 | 394.4 | **1.5×** |
| Int32 | 256 | 314.0 | 419.7 | 1.3× |
| Int32 | 1024 | 320.6 | 425.9 | 1.3× |
| Int32 | 4096 | 334.2 | 359.3 | 1.1× |
| Double | 1 | 16.7 | 181.9 | **10.9×** |
| Double | 4 | 47.0 | 168.0 | **3.6×** |
| Double | 16 | 120.0 | 310.0 | **2.6×** |
| Double | 64 | 223.1 | 425.3 | **1.9×** |
| Double | 256 | 315.1 | 469.8 | 1.5× |
| Double | 1024 | 358.9 | 477.1 | 1.3× |
| Double | 4096 | 373.8 | 483.3 | 1.3× |
| String | 1 | 8.0 | 59.2 | **7.4×** |
| String | 4 | 21.7 | 57.9 | **2.7×** |
| String | 16 | 41.3 | 60.7 | **1.5×** |
| String | 64 | 62.9 | 80.6 | 1.3× |
| String | 256 | 75.2 | 82.0 | 1.1× |
| String | 1024 | 79.4 | 84.0 | 1.1× |
| String | 4096 | 80.4 | 83.4 | 1.0× |

Small batches (1–16): **2×–11×**. Large batches: no regression.

When the columns in the variant are sparse, during compaction, the
frequent add calls on the zonemap index cause field creation and field
comparison to account for around 12% of CPU usage. After optimization,
this drops to 3%.

None

- Test <!-- At least one of them must be included. -->
    - [ ] Regression test
    - [ ] Unit Test
    - [ ] Manual test (add detailed scripts or steps below)
    - [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
        - [ ] Previous test can cover this change.
        - [ ] No code files have been changed.
        - [ ] Other reason <!-- Add your reason?  -->

- Behavior changed:
    - [ ] No.
    - [ ] Yes. <!-- Explain the behavior change -->

- Does this need documentation?
    - [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg:
apache/doris-website#1214 -->

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
(cherry picked from commit 555ea5c)
@csun5285 csun5285 requested a review from yiguolei as a code owner May 7, 2026 07:20
@hello-stephen
Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@csun5285
Copy link
Copy Markdown
Contributor Author

csun5285 commented May 7, 2026

run buildall

@csun5285
Copy link
Copy Markdown
Contributor Author

csun5285 commented May 7, 2026

/review

@csun5285 csun5285 changed the title [fix](zone-map) Avoid per-row Field temporaries in TypedZoneMapIndexWriter (#62544) [branch-4.1](zone-map) Avoid per-row Field temporaries in TypedZoneMapIndexWriter (#62544) May 8, 2026
@github-actions github-actions Bot added the approved Indicates a PR has been approved by one committer. label May 9, 2026
@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented May 9, 2026

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented May 9, 2026

PR approved by anyone and no changes requested.

@yiguolei yiguolei merged commit 7a5485f into apache:branch-4.1 May 9, 2026
28 of 33 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by one committer. reviewed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants