Skip to content

[FLINK] Support Apache Pulsar as DataSource#12297

Open
zhanglistar wants to merge 4 commits into
apache:mainfrom
zhanglistar:fix-flink-nexmark-window-aggregate
Open

[FLINK] Support Apache Pulsar as DataSource#12297
zhanglistar wants to merge 4 commits into
apache:mainfrom
zhanglistar:fix-flink-nexmark-window-aggregate

Conversation

@zhanglistar

@zhanglistar zhanglistar commented Jun 15, 2026

Copy link
Copy Markdown
Contributor

Summary

Add Gluten Flink planner/runtime support for Pulsar sources backed by Velox. Solves #12310 .
Depends on bigo-sg/velox#42 and bigo-sg/velox4j#35.

This patch:

  • Adds a PulsarSourceSinkFactory that maps Flink Pulsar table/source options to Velox Pulsar connector parameters.
  • Registers connector-pulsar in Gluten Flink connector config and service discovery.
  • Passes the planner classloader into runtime source/sink factory discovery so planner-side factories can be loaded reliably.
  • Makes blocked Velox source tasks wait instead of spinning.
  • Adds unit tests for Pulsar factory discovery, connector config registration, option mapping, and wrapped Pulsar source detection.

Test Plan

  • mvn -pl ut -am -Dtest=PulsarSourceSinkFactoryTest -DfailIfNoTests=false test
  • Manually verified a Flink SQL Pulsar source job with parallelism 2 against local Pulsar standalone:
    • job remained RUNNING
    • two Pulsar CPP consumers were connected under a Shared subscription
    • both Flink source subtasks consumed records
    • Pulsar backlog reached 0 for the test subscription

AI Tooling Disclosure

Cowork with codex.

Copilot AI review requested due to automatic review settings June 15, 2026 04:02
@github-actions github-actions Bot added the FLINK label Jun 15, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@zhanglistar zhanglistar changed the title Fix Flink Nexmark window aggregate tests [FLINK] Fix Flink Nexmark window aggregate tests Jun 15, 2026
@zhanglistar zhanglistar changed the title [FLINK] Fix Flink Nexmark window aggregate tests [FLINK] Support Apache Pulsar connector Jun 17, 2026
@zhanglistar zhanglistar changed the title [FLINK] Support Apache Pulsar connector [FLINK] Support Apache Pulsar as DataSource Jun 17, 2026
@zhanglistar zhanglistar force-pushed the fix-flink-nexmark-window-aggregate branch from 7814777 to 13a947e Compare June 17, 2026 03:32
Copilot AI review requested due to automatic review settings June 17, 2026 03:32

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@github-actions github-actions Bot added the INFRA label Jun 17, 2026
Copilot AI review requested due to automatic review settings June 17, 2026 09:06

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@zhanglistar

Copy link
Copy Markdown
Contributor Author

@KevinyhZou @liujiayi771 @lgbo-ustc Pls review.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants