Skip to content

[fix](be) Keep runtime filter merger inputs private on branch-4.1 - #70768

Draft
mrhhsg wants to merge 2 commits into
apache:branch-4.1from
mrhhsg:pick_68684_4_1
Draft

mrhhsg wants to merge 2 commits into
apache:branch-4.1from
mrhhsg:pick_68684_4_1

Conversation

@mrhhsg

@mrhhsg mrhhsg commented Nov 19, 2026

Copy link
Copy Markdown
Member

Summary

Backport the runtime-filter ownership fixes from #68684 and #70661 to branch-4.1. A producer wrapper can be consumed directly by a local target while its filter is also merged for another target. Reusing that wrapper in the merger lets later merges change a filter that the direct consumer is already probing.

  • Deep-copy the first merger input and the mutable IN, Bloom, and MinMax filter state. Copy an incoming Bloom filter before adding IN values during an IN-to-Bloom transition.
  • Count an input and mark the merged filter ready only after copying or merging succeeds, so a failed input cannot cause a partial filter to be published.
  • Preserve the existing local, broadcast, and remote delivery paths. Adapt the ownership and failed-merge unit tests to the branch-4.1 interfaces.

Validation

  • BUILD_TYPE=ASAN ./run-be-ut.sh --run -j 16 --filter='RuntimeFilterMergerTest.*:RuntimeFilterProducerTest.*:RuntimeFilterWrapperTest.*' — 44/44 passed.
  • BUILD_TYPE=ASAN ./run-be-ut.sh --run -j 16 --filter='*RuntimeFilter*:*BloomFilter*:*MinMax*:*SyncSize*-TableReaderTest.CountStarFallbackKeepsLateRuntimeFilterCarrierValues' — 291/291 passed.
  • ./build.sh --be -j 32 — passed.
  • (cd fe && mvn -o -pl fe-core -am -DskipTests validate) — passed.
  • doris-local-regression run -d query_p0/runtimefilterV2 -s rfv2 — 1/1 suite passed on a single-FE/single-BE cluster.
  • Changed C++ files passed clang-format --dry-run --Werror; repository format check and git diff --check passed.

Multi-BE RPC, mixed-version operation, and production Bloom-copy performance have not been validated locally.

…rapper when the filter is also merged apache#68684

Cherry-picked from apache#68684 (d357885).

Resolve branch-4.1 conflicts by retaining the existing compile-check and
runtime filter interfaces. Adapt the unit tests to the branch's expression
type and omit assertions for the master-only bucket-prune cache; keep the
wrapper ownership and filter-value checks.

Tests: ./run-be-ut.sh -j 40 --run --filter='RuntimeFilterMergerTest.*:RuntimeFilterProducerTest.*:RuntimeFilterWrapperTest.*'

(cherry picked from commit d357885)
Backport the ownership simplification and arrival-count fix from apache#70661
(8d5b213, 4f12834, ab348b3, 1e3335d).

Always clone the first merger input and the Bloom filter used during an
IN-to-Bloom transition, so callers do not need to prove exclusive ownership.
Record an input arrival and mark the merger ready only after accumulation
succeeds. Keep local and remote filter delivery unchanged.

Adapt the failed-merge test to branch-4.1 with a deterministic Bloom-size
mismatch, since its Bloom directory allocation bypasses the memory tracker.

Tests: focused ASAN BE runtime-filter UT, broad RF-related ASAN BE UT,
rfv2 SQL regression, C++ format checks, and FE checkstyle.
@mrhhsg

mrhhsg commented Nov 19, 2026

Copy link
Copy Markdown
Member Author

run buildall

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant