Skip to content

[fix](variant) Read shredded scalar paths without rebuilding the root - #70767

Open
morningman wants to merge 1 commit into
apache:masterfrom
morningman:b2m-t22-variant-shredded-leaf
Open

morningman wants to merge 1 commit into
apache:masterfrom
morningman:b2m-t22-variant-shredded-leaf

Conversation

@morningman

Copy link
Copy Markdown
Contributor

Background

Parquet can store Variant fields as typed columns alongside encoded residual values. Doris can project individual fields, using the root metadata dictionary to decode residual keys. Row-group statistics can prove that residual columns are entirely NULL.

What problem does this PR solve?

Issue Number: None
Related PR: #66941 (same change on 4.1/4.2), #68597 (STRING profile stabilization), #66413 (introduced the reader), #69016 (deep leaf projection), #70664 (scan splitting)
PR Type: Bugfix

Problem Summary:

This ports #66941 and #68597 to master. Complete STRING leaves and mixed residual batches unnecessarily rebuild Variant roots; a NULL typed root can also hide a requested value stored in its residual. Resolve rows from their typed leaf or the first residual that owns the path, and prune unused physical columns. Most added lines are tests.

Original authors: @hubgeter and @Gabriel39.

Details

  • Change map: Production +682/-156 lines in 9 files; tests +1400/-88 in 8 files. The production parts are leaf/residual decoding and validation caching, mapper/row-group projection and byte accounting, and profile/composite validation. No generated outputs or docs.
  • Frequency: Deterministic for the affected complete STRING, mixed-residual and NULL typed-root shapes.
  • Root cause: The path walk ignores the root residual. Complete-state scalar guards and batch-wide residual rejection force fallback; the mapper excludes scalar identities the reader can normalize.
  • Why existing tests missed it: Existing BE tests exercised reconstructed values and conservative fallback. Iceberg/Paimon regressions checked values and BIGINT projection, without complete STRING direct access, row-level residual ownership, shared validation or pruned physical-column assertions.
  • Fix: Carry metadata and terminal residuals with scalar projections, then finalize them per row group. Preserve typed UTF-8 leaves; normalize other scalar identities through the Parquet schema. Cache dictionary field IDs and residual validation, and validate every composite segment. Preserve master's UUID/Decimal128 rules and shared profile ownership.
  • Other paths and branches: Iceberg and native Paimon share this reader; both suites are covered. Complex/repeated projections retain complete fallback. Branches 4.1/4.2 already have the change. Local 3.1/4.0 refs have no V2 reader here and were not refreshed. Statistics filtering retains its conservative ancestor-residual rule.
  • Behavior change: Add VariantDirectLeafResidualMergedRows, VariantResidualSeekRows and VariantResidualProjectionRowGroupColumns to profiles. Configs and defaults are unchanged.

Testing

  • Local macOS arm64, LLVM22, ASAN: 494 cases across the six BE files; 491 passed, two optional microbenchmarks skipped, one preexisting comparison test failed. All 28 newly ported cases passed without skips. A temporary focused test build included their support sources and was restored afterward.
  • Restoring master's original lookup/composite methods makes four new controls fail at the intended assertions. The existing TypedComparisonBoundariesMatchEncodedWithoutMaterializing repeats twelve sign-versus-magnitude failures with those original methods; it is left unchanged for a separate test-only fix.
  • Original Iceberg and Paimon Variant suites both passed locally, each with zero skipped scripts, using Spark 4, MinIO and Iceberg REST with SQLite. The 4096-row STRING query reports one leaf projection, zero full/residual projections and zero reconstructed rows. No timing speedup claim.

Release note

None

Check List (For Author)

  • Behavior changed: Yes (see Details)
  • Does this need documentation: No
  • Pre-submission status: Partially verified: the existing macOS comparison assertion fails independently; full Linux BE/unit validation was not run.

AI Disclosure

  • Tool: Codex (GPT-6)
  • Generated: adaptation, tests, verification and this description; the main implementation and tests originate from the related PRs.
  • Human review: not stated

### Background

Parquet can store Variant fields as typed columns alongside encoded residual values. Doris can project individual fields, using the root metadata dictionary to decode residual keys. Row-group statistics can prove that residual columns are entirely NULL.

### What problem does this PR solve?

Issue Number: None
Related PR: apache#66941 (same change on 4.1/4.2), apache#68597 (STRING profile stabilization), apache#66413 (introduced the reader), apache#69016 (deep leaf projection), apache#70664 (scan splitting)
PR Type: Bugfix

Problem Summary:

This ports apache#66941 and apache#68597 to master. Complete STRING leaves and mixed residual batches unnecessarily rebuild Variant roots; a NULL typed root can also hide a requested value stored in its residual. Resolve rows from their typed leaf or the first residual that owns the path, and prune unused physical columns. Most added lines are tests.

Original authors: @hubgeter and @Gabriel39.

### Details

- **Change map:** Production +682/-156 lines in 9 files; tests +1400/-88 in 8 files. The production parts are leaf/residual decoding and validation caching, mapper/row-group projection and byte accounting, and profile/composite validation. No generated outputs or docs.
- **Frequency:** Deterministic for the affected complete STRING, mixed-residual and NULL typed-root shapes.
- **Root cause:** The path walk ignores the root residual. Complete-state scalar guards and batch-wide residual rejection force fallback; the mapper excludes scalar identities the reader can normalize.
- **Why existing tests missed it:** Existing BE tests exercised reconstructed values and conservative fallback. Iceberg/Paimon regressions checked values and BIGINT projection, without complete STRING direct access, row-level residual ownership, shared validation or pruned physical-column assertions.
- **Fix:** Carry metadata and terminal residuals with scalar projections, then finalize them per row group. Preserve typed UTF-8 leaves; normalize other scalar identities through the Parquet schema. Cache dictionary field IDs and residual validation, and validate every composite segment. Preserve master's UUID/Decimal128 rules and shared profile ownership.
- **Other paths and branches:** Iceberg and native Paimon share this reader; both suites are covered. Complex/repeated projections retain complete fallback. Branches 4.1/4.2 already have the change. Local 3.1/4.0 refs have no V2 reader here and were not refreshed. Statistics filtering retains its conservative ancestor-residual rule.
- **Behavior change:** Add `VariantDirectLeafResidualMergedRows`, `VariantResidualSeekRows` and `VariantResidualProjectionRowGroupColumns` to profiles. Configs and defaults are unchanged.

### Testing

- Local macOS arm64, LLVM22, ASAN: 494 cases across the six BE files; 491 passed, two optional microbenchmarks skipped, one preexisting comparison test failed. All 28 newly ported cases passed without skips. A temporary focused test build included their support sources and was restored afterward.
- Restoring master's original lookup/composite methods makes four new controls fail at the intended assertions. The existing `TypedComparisonBoundariesMatchEncodedWithoutMaterializing` repeats twelve sign-versus-magnitude failures with those original methods; it is left unchanged for a separate test-only fix.
- Original Iceberg and Paimon Variant suites both passed locally, each with zero skipped scripts, using Spark 4, MinIO and Iceberg REST with SQLite. The 4096-row STRING query reports one leaf projection, zero full/residual projections and zero reconstructed rows. No timing speedup claim.

### Release note

None

### Check List (For Author)

- Behavior changed: Yes (see Details)
- Does this need documentation: No
- Pre-submission status: Partially verified: the existing macOS comparison assertion fails independently; full Linux BE/unit validation was not run.

### AI Disclosure

- Tool: Codex (GPT-6)
- Generated: adaptation, tests, verification and this description; the main implementation and tests originate from the related PRs.
- Human review: not stated

Co-authored-by: daidai <changyuwei@selectdb.com>
Co-authored-by: Gabriel <liwenqiang@selectdb.com>
@morningman

Copy link
Copy Markdown
Contributor Author

run buildall

@morningman

Copy link
Copy Markdown
Contributor Author

/review

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No actionable production finding identified at 9e1d486. Reviewed all 17 changed files and the affected caller chains. No existing inline threads or additional user focus points were supplied; live comments/reviews were also empty when refreshed.

Critical checkpoint conclusions (static inspection):

  • Goal and evidence: the code enables annotated STRING direct leaves, row-level terminal residual merging, shared validation, and per-row-group physical pruning. Added BE cases and Iceberg/Paimon profile assertions target those behaviors; their execution was not independently verified.
  • Scope and clarity: the nine production files and eight test files stay focused on Variant projection, materialization, and observability.
  • Concurrency and locks: no production threads or lock order are added. Existing materialization mutexes protect caches; scanning/decoding occurs outside the cache lookup/publication locks. COW isolates mutable appends from shared states.
  • Lifecycle and memory: immutable physical schemas and columns own retained views; selections create new states, append invalidates caches, allocator-aware residual bookkeeping is accounted, and new persistent counters use shared ownership. No new cross-TU production initialization dependency was found.
  • Configuration: no settings, defaults, or dynamic-reload behavior change.
  • Compatibility: Parquet annotations remain in retained descriptors; binary/UUID/temporal/narrow-decimal leaves normalize through their schema. No storage or block-wire layout change was found. Ancestor residual omission follows the Parquet Objects contract: objects require a present typed object, and residuals exclude shredded fields.
  • Generated protocols: no Thrift/Protobuf definitions or field IDs change.
  • Parallel paths: Iceberg and native Paimon share the reader and both regression suites are updated; complex/repeated cases retain complete fallback, and ordinary non-Variant projections retain their existing path.
  • Conditional and error paths: null typed carriers, missing keys, incompatible carrier combinations, metadata validation, scalar identities, and fallback branches were traced through extraction. Every composite segment is visited before normalized fallback; no new silently ignored Status was identified.
  • Tests: inspected the mapper, row-group planner, statistics, core composite, reader, and external integration additions, including negative carrier/metadata cases, selected state ownership, and STRING no-copy/profile checks. Runtime coverage is unproven in this review environment.
  • Expected outputs: no generated regression .out files change; changed assertions align with the inspected projection and value semantics.
  • Observability: residual merge/seek and residual projection counters follow existing profile ownership; physical column sets used for byte accounting, indexes, native readers, and prefetch agree with rewritten row-group requests.
  • Transactions and persistence: no transaction, EditLog, snapshot-selection, or persistence protocol is modified.
  • Data writes and atomicity: no write path changes; scan state resets between row groups and preserves the existing selection/lazy-reader boundaries.
  • FE/BE propagation: no new cross-process variable or request field requires forwarding changes.
  • Performance: inspected pruning and caching avoid unrelated columns and repeated carrier validation; this review establishes no timing speedup, allocation benchmark, or final ELF/glibc compatibility result.
  • Other concerns and limits: a mixed object/array multi-segment helper path can reject an array suffix inside a residual. It is excluded from inline findings because exhaustive production caller search finds only one-segment extraction; current SQL chains extract encoded prefixes before indexing. A future fused multi-segment caller needs coverage of this helper boundary.

Validation was static only: no builds, tests, benchmarks, service execution, source edits, or submodule initialization. The PR's reported existing comparison-test failure and absence of a full Linux validation run were not independently reproduced. Review convergence is assessed separately from runtime validation.

Review converged after 2 round(s).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant