Skip to content

feat(connectors): add Apache Airflow trigger sink#3716

Open
avirajkhare00 wants to merge 5 commits into
apache:masterfrom
avirajkhare00:feat/connectors-airflow-trigger-sink
Open

feat(connectors): add Apache Airflow trigger sink#3716
avirajkhare00 wants to merge 5 commits into
apache:masterfrom
avirajkhare00:feat/connectors-airflow-trigger-sink

Conversation

@avirajkhare00

Copy link
Copy Markdown
Contributor

Summary

  • Add iggy_connector_airflow_sink that triggers Airflow DAG runs (POST /api/v1/dags/{dag_id}/dagRuns) from Iggy messages
  • Support auth modes none / basic / bearer, optional per-message dag_id header override, and nested conf.iggy metadata
  • Treat HTTP 409 as success for deterministic dag_run_id redelivery; permanent 4xx drop without retry
  • WireMock integration test plus example config / README; tracked by feat(connectors): add Apache Airflow trigger connector #3715

Test plan

  • cargo fmt --all
  • cargo sort --no-format --workspace
  • cargo clippy -p iggy_connector_airflow_sink --all-targets -- -D warnings
  • cargo test -p iggy_connector_airflow_sink (11 unit tests)
  • cargo test -p integration -- connectors::airflow with WireMock (Docker)
  • Local e2e against real Airflow 2.10.5 standalone in Docker (3 DAG runs created)

Add a sink plugin that consumes Iggy messages and creates Airflow DAG
runs via the stable REST API, with basic/bearer auth, deterministic
dag_run_id for idempotent retries, and WireMock integration tests.

Closes apache#3715
@github-actions

Copy link
Copy Markdown

Thanks for the PR. It is labeled S-waiting-on-review and queued for review.

Slash commands (own line, regular comment) move it around the queue:

  • /ready - back to S-waiting-on-review after addressing feedback
  • /author - flip to S-waiting-on-author while you finish changes
  • /request-review @user-or-team - request a reviewer

See CONTRIBUTING.md for details.

@github-actions github-actions Bot added the S-waiting-on-review PR is waiting on a reviewer label Jul 21, 2026
Adjust table separators and link bare URLs so MD060/MD034 pass.
@codecov

codecov Bot commented Jul 21, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 76.77165% with 118 lines in your changes missing coverage. Please review.
✅ Project coverage is 41.31%. Comparing base (4a6f173) to head (daa2ec6).

Files with missing lines Patch % Lines
core/connectors/sinks/airflow_sink/src/lib.rs 76.77% 106 Missing and 12 partials ⚠️
Additional details and impacted files
@@              Coverage Diff              @@
##             master    #3716       +/-   ##
=============================================
- Coverage     74.31%   41.31%   -33.01%     
  Complexity      937      937               
=============================================
  Files          1303     1302        -1     
  Lines        148600   129447    -19153     
  Branches     124041   104963    -19078     
=============================================
- Hits         110427    53475    -56952     
- Misses        34705    73136    +38431     
+ Partials       3468     2836      -632     
Components Coverage Δ
Rust Core 33.56% <76.77%> (-41.01%) ⬇️
Java SDK 62.44% <ø> (ø)
C# SDK 71.05% <ø> (-1.14%) ⬇️
Python SDK 92.15% <ø> (ø)
PHP SDK 84.29% <ø> (ø)
Node SDK 92.49% <ø> (-0.02%) ⬇️
Go SDK 42.87% <ø> (ø)
Files with missing lines Coverage Δ
core/connectors/sinks/airflow_sink/src/lib.rs 76.77% <76.77%> (ø)

... and 435 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

cargo-machete failed CI because tokio was declared but not used.
finalize_pr failed solely because LowLevelE2E_Client.GetClientsReflectsSessionRemovalAfterDisconnect failed (278 passed, 1 failed). No C++ changes in this PR.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

S-waiting-on-review PR is waiting on a reviewer

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant