Skip to content

[fix](external) Preserve sampling and backend placement with coarse splits - #70746

Open
morningman wants to merge 2 commits into
apache:masterfrom
morningman:fix/external-sampling-be-placement
Open

morningman wants to merge 2 commits into
apache:masterfrom
morningman:fix/external-sampling-be-placement

Conversation

@morningman

@morningman morningman commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

Background

The frontend (FE) plans external-file byte ranges and assigns them to backends (BEs). Native Parquet and ORC readers can refine a range into physical tasks, which remain on its assigned BE. Hive TABLESAMPLE selects from the FE-planned ranges. Parquet files without physical columns can still supply footer row counts for partition-column projections.

What problem does this PR solve?

Issue Number: None
Related PR: #70664 (introduced the master regressions), #66802 (earlier release-branch implementation), #65674 (introduced native Parquet partial-range selection)
PR Type: Bugfix

Problem Summary:

Hive sampling can select an entire large file, and coarse synchronous scans can leave available BEs idle. Keep legacy FE sizing for applied samples and subdivide eligible ranges before backend assignment, within existing count and size budgets. Give Parquet files without physical columns a unique range owner so subdivision preserves partition-only queries. Most of the diff tests sampling, assignment, metadata retention and row ownership.

The FE planning fixtures use a single 400 MiB file:

Scenario Without the fix With the fix
4% sample with legacy 16 MiB ranges 400 MiB selected range 16 MiB selected range
Two eligible BEs with round-robin assignment 1 BE receives ranges 2 BEs receive ranges

Details

  • Change map: 88 production lines (+85/-3) in five files and 400 test lines (+397/-3) in five files; no generated output. Production parts, in reading order: legacy sampling sizing (3 lines); bounded FE subdivision and metadata-preserving copies (78 lines); root-only Parquet ownership (7 lines, follow-up commit 4431620159a). Tests account for 82% of added lines.
  • Frequency: Deterministic in FE fixtures with sampling or too few ranges for available BEs, and in BE fixtures for root-only partial ranges.
  • Root cause: Coarse sizing runs before FE sampling, and BE refinement cannot move children across BEs. Parquet partial-range selection additionally requires a column-chunk midpoint that root-only files cannot provide.
  • Why existing tests missed it: Existing FE tests cover sampling, sizing and placement separately. The Hive sampling regression allows a full-file sample; splitting tests omit placement assertions. The root-only reader test uses a whole-file range.
  • Fix: Omit coarse hints for applied samples. Split underfilled synchronous plans up to available BEs and max_file_split_num, keeping children at least the BE target size. Assign root-only Parquet groups to the nonempty range containing byte zero; preserve ordinary midpoint ownership and missing-chunk errors for physical schemas.
  • Other paths and branches: Native plugin scans and TVFs share planning; existing exclusions and lazy batch/streaming assignment remain. Branches 4.1/4.2 need adapted placement backports and carry the Parquet guard; their sampling precedes splitting. Branches 3.1/4.0 lack these paths. No SPI, wire or storage-format change.
  • Behavior change: Underfilled native plans can expose more FE ranges and scanning BEs in EXPLAIN/profile output, within existing budgets.

Testing

  • Local FE run on bb55986f9f4: 179 tests passed. The two original reproductions failed on unmodified master and with the fix entry points reverted.
  • Local FE run on the current head: 30 tests passed, including a 101-byte file split into [0,50) and [50,101) with BE target=1 and retained partition values.
  • Local BE baseline: 6 related tests ran; the 2 new cases failed with Parquet row group 0 has no column chunks, while the whole-file partition-projection control succeeded. With the fix, all 7 related BE tests passed, covering adjacent/empty ranges, exact row-group selection, malformed missing chunks and ordinary physical-column ranges. The real TableReader fixture returns five p=7 rows exactly once across the partial ranges.
  • Extended external_table_p0/hive/test_two_level_file_splits with repeatable PERCENT and ROWS comparisons against its legacy reference; the external suite was not run locally.

Notes for Reviewers

FE reuses planned ranges and connector descriptors, without another remote enumeration. BE resolves root-only ownership from the physical schema. Placement counts do not establish latency gains.

The standard BE runner used a temporary selection of related test sources and required helpers after full ASAN linking exhausted local disk. Production libraries retained normal ASAN flags; all temporary build-file and macOS test-format workarounds were restored. The full BE UT executable remains unverified.

Release note

Preserve Hive sampling granularity and useful multi-BE placement when coarse FE splitting is enabled. Parquet files without physical columns remain readable after FE subdivision.

Check List (For Author)

  • Behavior changed: Yes (see Details)
  • Does this need documentation: No
  • Pre-submission status: Partially verified: the full BE UT executable, external Hive suite and live multi-BE execution were not verified.

AI Disclosure

…plits

### Background

The frontend (FE) plans external-file byte ranges and assigns each range to a backend (BE). Native Parquet and ORC readers can refine a range into physical tasks, which remain on that BE. Hive `TABLESAMPLE` selects a subset of the ranges planned by FE.

### What problem does this PR solve?

Issue Number: None
Related PR: apache#70664 (introduced the master regressions), apache#66802 (earlier release-branch implementation)
PR Type: Bugfix

Problem Summary:

Hive sampling can select an entire large file, and coarse synchronous scans can leave available BEs idle. Preserve legacy FE sizing for applied samples and subdivide eligible ranges before assignment when coarse planning leaves too few ranges, within existing count and size budgets. Most of the diff tests sampling, actual FE placement and metadata retention.

The FE planning tests use a single 400 MiB file:

| Scenario | Without the fix | With the fix |
|---|---|---|
| 4% sample with legacy 16 MiB ranges | 400 MiB selected range | 16 MiB selected range |
| Two eligible BEs with round-robin assignment | 1 BE receives ranges | 2 BEs receive ranges |

### Details

- **Change map:** 81 production lines in four files and 273 test lines in three files; no generated output. The sampling gate accounts for three production lines; bounded placement and metadata-preserving range copies account for the other 78.
- **Frequency:** Reproduced deterministically in FE planning tests with coarse splitting enabled, for applied Hive sampling or too few native ranges for the available BEs.
- **Root cause:** The connector receives the coarse hint before FE applies sampling. Assignment then treats each remaining coarse range as indivisible, while BE refinement only publishes local children.
- **Why existing tests missed it:** `PluginDrivenScanNodeTableSampleTest` supplies prebuilt ranges; sizing and backend-policy tests cover their parts separately. The Hive sampling regression allows a full-file sample, and the two-level splitting regression checks results without asserting placement.
- **Fix:** Omit the coarse hint for applied samples. For underfilled synchronous plans, split the largest eligible native ranges up to the available BE count and `max_file_split_num`, stopping before children fall below the BE target. Copies retain connector metadata and proportional scheduling weights; planning does not repeat remote file enumeration.
- **Other paths and branches:** The shared path covers native plugin scans and file TVFs, retaining the serial, metadata COUNT, format/capability, compression and HTTP gates. Streaming and batch producers retain lazy assignment. Branches 4.1/4.2 have analogous placement loss in their legacy planners and need adapted backports; their file-level sampling precedes splitting. Branches 3.1/4.0 lack coarse FE splitting. No SPI, wire or storage-format change is required.
- **Behavior change:** Underfilled native plans may expose more FE ranges and scanning BEs in EXPLAIN/profile output, within the existing budgets.

### Testing

- Local `run-fe-ut.sh`: 179 tests in 10 related FE/connector classes passed, with zero failures, errors or skips. Coverage includes sampling, actual assignment/Thrift serialization, range bounds, serial/count exclusions and metadata/weight retention.
- The initial unmodified-master run failed both reproductions. Reverting the two fix entry points with the final tests again selected 400 MiB instead of 16 MiB and assigned the scan to only one BE; restoring them passes.
- Extended `external_table_p0/hive/test_two_level_file_splits` with repeatable PERCENT and ROWS comparisons against its legacy reference. The external Hive suite was not run locally.

### Notes for Reviewers

The fallback reuses already planned ranges and their immutable connector descriptors. It adds no remote planning pass or cross-BE child transfer. The BE target also bounds the minimum FE child size in this fallback; placement counts do not imply a fixed latency improvement.

### Release note

Preserve Hive sampling granularity and useful multi-BE placement when coarse FE splitting is enabled.

### Check List (For Author)

- Behavior changed: Yes (see Details)
- Does this need documentation: No
- Pre-submission status: Partially verified: the external Hive suite and live multi-BE execution were not run.

### AI Disclosure

- Tool: Codex (GPT-6)
- Generated: Code, tests and this description.
- Human review: not stated
- Prompts: Analyze 924060929's comments on apache#70664, then fix both issues from the latest master after that PR merged and open a separate PR.
@morningman

Copy link
Copy Markdown
Contributor Author

/review

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed all seven changed paths at bb55986f9f43df3c8ca95682a30fe2b65caaca4c. One new P2 finding: FE subdivision can turn a supported whole-file Parquet scan with no physical column chunks into partial ranges that fail native range selection. No existing inline findings were present to reconfirm or duplicate.

The reviewed execution flow is:

getSplits: connector sizing -> wrap ranges -> optional TABLESAMPLE
createScanRangeLocations: bounded FE subdivision -> backend assignment -> Thrift ranges
BE: TableReader projection -> Parquet/ORC range ownership -> optional local physical refinement

Critical checkpoint conclusions:

  • Goal and evidence: applied Hive samples retain legacy sizing; ordinary underfilled synchronous native plans gain bounded FE ranges before assignment. The new FE tests inspect sampling, actual round-robin owners, serialized coordinates, bounds and metadata. The inline finding identifies an exception to native range safety.
  • Scope and reuse: four focused production files reuse planned ranges and connector descriptors; tests cover the changed entry points. No second remote enumeration is introduced.
  • Concurrency and locking: the new queue, list and copies are local to synchronous planning before assignment. Batch producers remain in their separate lazy path. No new shared mutation, lock, lock ordering or work inside locks was introduced.
  • Lifecycle: children retain file-level immutable descriptors; no new executor, resource lifetime, cross-BE shared reader or static initialization dependency was added.
  • Configuration: no new settings. Existing FE/BE targets, positive max_file_split_num, scanner serial mode and small-LIMIT gates are read during planning. The finding includes a permitted small positive BE target.
  • Compatibility: no SPI, wire, storage-format or function-registration change. Existing native range serialization is reused; copied start/length reach Thrift without serializer overwrites.
  • Generated definitions: no Thrift or protobuf files changed; field-ID review is not applicable.
  • Parallel paths: synchronous plugin scans and file TVFs share the fallback; RemoteDoris remains nonnative. Applied TABLESAMPLE forces synchronous planning; batch/streaming retains lazy assignment. HTTP TVFs preserve their exclusion.
  • Conditions: capability, actual format, compression, precomputed count, count-star, serial and minimum-child-size gates have concrete purposes. These gates do not exclude the root-only Parquet case in the inline finding.
  • Test coverage: added cases cover both sampling modes, counts/size limits, nonzero offsets, unsupported formats/capabilities, count and compressed exclusions, parent immutability, partition metadata and weights. The regression additions compare ordered, repeatable samples against a legacy reference. They omit the reported no-column-chunk FE-to-BE path.
  • Test results: no generated result files changed. Assertions and expected results were statically inspected; author-reported test passes were not independently reproduced.
  • Observability: existing split/backend counts, Thrift coordinates, EXPLAIN and assigned-weight profile remain available; no new hot-path logging was added.
  • Transactions and persistence: unchanged; no journal, replay, failover or visible-version behavior was added.
  • Data writes and atomicity: unchanged; this is query-range planning and introduces no write/crash-recovery state.
  • FE-to-BE variables: none added. Existing file-size, modification-time, partition and table-format metadata, plus range coordinates and target size, remain the protocol.
  • Performance: subdivision adds at most the bounded desired FE range count, with O(B log B) work and O(B) objects for backend bound B. It does no remote I/O. Physical unit count and configured hash/locality policies can still limit useful distinct owners; round-robin placement does not establish a latency improvement.
  • Other risks and user focus: descriptor sharing, scheduling identity, proportional weights, sample repeatability/reuse, ordinary half-open native ownership and file-global Paimon deletion-vector positions were traced. No additional user focus was supplied.

Validation limits: static inspection only. No builds, tests, benchmarks, live external Hive execution, rolling-upgrade run or multi-BE runtime experiment was performed. No source-level question remains unresolved after the review rounds; these execution limits remain.

Review converged after 2 round(s).

result.add(largest);
break;
}
candidates.add(largest.withRange(largest.getStart(), half));

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Keep root-only Parquet ranges whole

This can turn a supported partition-only scan into a corruption error. For example, with two BEs, synchronous Hive planning and file_split_size_on_be=1 (an accepted session value), SELECT p FROM t over a positive-row, root-only Parquet file used to receive one full-file range. The partition projection becomes a constant, so the native reader can return the footer row count with no physical columns (see PositiveRowRootOnlyFileKeepsInitializedReaderForCount). These copies instead produce two partial ranges; select_native_row_groups_by_scan_range() rejects both with Parquet row group ... has no column chunks. BE physical refinement deliberately retains the initialized parent for this case because it has no byte envelope. Preserve that whole-file behavior here, or teach partial-range ownership to handle such row groups, and cover the partition-only query through FE subdivision.

### Background

FE assigns external-file byte ranges to BEs. The native Parquet reader normally assigns row groups by their column-chunk midpoint. A root-only Parquet file has no physical columns but can still supply footer row counts for partition-column projections.

### What problem does this PR solve?

Issue Number: None
Related PR: apache#70746 (introduces the FE subdivision entry point), apache#65674 (introduced the native partial-range guard)
PR Type: Bugfix

Problem Summary:

FE subdivision can make a supported partition-only Parquet scan fail with `Parquet row group 0 has no column chunks`. Assign root-only row groups exclusively to the nonempty range containing byte zero. In the BE fixture, the whole file returns five rows with p=7; before the fix, its first partial range fails, and after the fix the two partial ranges return the same five rows once.

### Details

- **Frequency:** Always for a root-only file passed through the partial-range selector. With two BEs, a one-byte BE target makes FE subdivision reachable for a small file.
- **Root cause:** Partial ranges require a physical column-chunk midpoint, while the whole-file path already supports root-only row counts.
- **Why existing tests missed it:** The root-only reader test uses a whole-file range. Existing FE placement and Hive splitting tests lack a root-only partition projection across partial ranges.
- **Fix:** Give root-only groups one byte-range owner. Preserve ordinary midpoint ownership, explicit row-group selection and corruption errors for missing chunks in a schema with physical columns.
- **Other paths and branches:** Native Parquet callers share this helper; ORC has stripe offsets. Branches 4.1/4.2 carry the same native guard; 3.1/4.0 lack this V2 helper. This task changes the master PR only.
- **Behavior change:** Valid root-only partial scans return their rows instead of a corruption error. No configuration, SPI, wire or storage-format change.

### Testing

- Final baseline tests: 6 BE tests ran; the 2 new cases failed with the reported no-column-chunk corruption, and 4 passed.
- With the fix: 7 related BE tests passed, including real TableReader partition materialization, full-file controls, adjacent and empty ranges, invalid missing chunks, and ordinary physical-column range selection.
- Standard FE runner: 30 tests passed, including the new 101-byte/one-byte-target FE assignment and serialization case. Checkstyle, clang-format v16, changed-line clang-tidy and BE build hygiene passed.

### Notes for Reviewers

Local BE runs used the standard runner with a temporary selection of related test sources and required test helpers because linking the full ASAN test executable exhausted disk. The production libraries used normal ASAN flags. All temporary source selections and the unrelated macOS printf workaround were restored before delivery.

### Release note

Prevent corruption errors when FE subdivision splits Parquet files without physical columns.

### Check List (For Author)

- Behavior changed: Yes (see Details)
- Does this need documentation: No
- Pre-submission status: Partially verified: the full BE UT executable and live external Hive/multi-BE execution were not verified.

### AI Disclosure

- Tool: Codex (GPT-6)
- Generated: Code, tests and this description.
- Human review: not stated
- Prompts: Analyze the comments on apache#70746 and fix confirmed defects.
@morningman

Copy link
Copy Markdown
Contributor Author

/review

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed all 10 changed paths at head 4431620 against the authoritative merge-base diff. No new actionable defect was verified. The FE subdivision queue produces contiguous, bounded half-open ranges and preserves file, partition, connector, schema/delete, identity, and scheduling metadata; eligibility gates retain compressed, count-pushed, non-native, serial, and unsupported paths. TABLESAMPLE disables connector coarse planning and precomputed COUNT(*) pushdown, forces synchronous sampling, and any later subdivision only partitions the already-selected byte ranges. The Parquet root-only exception is narrowly gated to a one-element zero-child schema, assigns rows only to the first nonempty range, and leaves malformed physical-schema row groups on the existing corruption path. The existing P2 thread 4625789929 covers the motivating root-only subdivision mechanism and is not duplicated; the current BE change addresses that case. The supplied review focus has no additional items. Static inspection covered the changed FE/BE callers, Hive/Iceberg/Paimon/Hudi native metadata paths, Paimon deletion-vector row positions, split sizing/assignment, and all added tests. No builds, tests, benchmarks, or live external scans were run, per the static-review contract. Review converged after round 1; no remaining source-level coverage gap was found.

Review converged after 1 round(s).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants