Skip to content

fix(connectors): Harden Elasticsearch sink test readiness#3729

Open
ryerraguntla wants to merge 4 commits into
apache:masterfrom
ryerraguntla:fix/elasticsearch-sink-test-flakiness
Open

fix(connectors): Harden Elasticsearch sink test readiness#3729
ryerraguntla wants to merge 4 commits into
apache:masterfrom
ryerraguntla:fix/elasticsearch-sink-test-flakiness

Conversation

@ryerraguntla

@ryerraguntla ryerraguntla commented Jul 22, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Fixes flaky Elasticsearch sink integration test where iggy-connectors
    never becomes healthy because sink open() can hang on Elasticsearch HTTP
    with no client timeout, while the connectors HTTP API only binds after
    plugin open.
  • Hardens the Elasticsearch Testcontainers fixture: use 127.0.0.1, always
    re-check /_cluster/health on reuse attach, self-heal a wedged
    iggy-test-elasticsearch container only, and sweep stale harness indices
    older than 30 minutes.
  • Improves harness readiness: poll connectors /health and fail fast if the
    child process dies during retries; refresh the index before document-count
    polls in the sink fixture.

Closes #3728

Test plan

  • cargo fmt --all
  • cargo sort --no-format --workspace
  • cargo clippy -p iggy_connector_elasticsearch_sink -p integration --all-features --all-targets -- -D warnings
  • cargo test -p integration -- connectors::elasticsearch::elasticsearch_sink::elasticsearch_sink_stores_json_messages
  • Confirm docker rm -f recovery targets only iggy-test-elasticsearch
    (no other Elasticsearch containers)
  • Optional: leave a reused iggy-test-elasticsearch running and re-run
    the sink test to verify reuse + readiness path

Notes

  • Container force-remove uses the fixed harness name
    iggy-test-elasticsearch only.
  • Stale index sweep is best-effort and age-gated so concurrent harness
    tests sharing the reused container are not disrupted.

Add a timeout to the Elasticsearch sink transport, make connectors-runtime readiness check `/health`, and strengthen the Elasticsearch test fixture startup path with retry, cluster-health polling, and stale-index cleanup. Also refresh the sink index before counting documents so test assertions see newly indexed data sooner.
Use short-timeout probes without retries for cluster health and stale
index sweep, and cap docker rm so a wedged daemon cannot stall setup.
@github-actions

Copy link
Copy Markdown

Thanks for the PR. It is labeled S-waiting-on-review and queued for review.

Slash commands (own line, regular comment) move it around the queue:

  • /ready - back to S-waiting-on-review after addressing feedback
  • /author - flip to S-waiting-on-author while you finish changes
  • /request-review @user-or-team - request a reviewer

See CONTRIBUTING.md for details.

@github-actions github-actions Bot added the S-waiting-on-review PR is waiting on a reviewer label Jul 22, 2026
@codecov

codecov Bot commented Jul 22, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 37.82%. Comparing base (a5001d3) to head (72fd3b7).

Additional details and impacted files
@@              Coverage Diff              @@
##             master    #3729       +/-   ##
=============================================
- Coverage     74.34%   37.82%   -36.53%     
  Complexity      950      950               
=============================================
  Files          1303     1301        -2     
  Lines        148712   129057    -19655     
  Branches     124225   104570    -19655     
=============================================
- Hits         110562    48811    -61751     
- Misses        34673    77640    +42967     
+ Partials       3477     2606      -871     
Components Coverage Δ
Rust Core 29.02% <100.00%> (-45.56%) ⬇️
Java SDK 62.64% <ø> (ø)
C# SDK 72.28% <ø> (ø)
Python SDK 92.27% <ø> (ø)
PHP SDK 84.52% <ø> (ø)
Node SDK 92.70% <ø> (ø)
Go SDK 43.08% <ø> (ø)
Files with missing lines Coverage Δ
...ore/connectors/sinks/elasticsearch_sink/src/lib.rs 67.60% <100.00%> (+1.42%) ⬆️

... and 515 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@ryerraguntla ryerraguntla changed the title fix(connectors): harden Elasticsearch sink test readiness fix(connectors): Harden Elasticsearch sink test readiness Jul 22, 2026
@ryerraguntla

Copy link
Copy Markdown
Contributor Author

/request-review @user-or-team

@slbotbm slbotbm left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

`

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

start() force-removes the shared container after any try_start() error, not only after confirming it is wedged. Transient health, port-inspection, or Docker errors can therefore remove an Elasticsearch container still used by another process. The fixed name limits which container is removed but does not provide ownership or concurrency safety. Could this recovery path verify the container is unhealthy and coordinate removal with a cross-process lock before running docker rm -f?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let me add remove when required.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed

@github-actions github-actions Bot added S-waiting-on-author PR is waiting on author response and removed S-waiting-on-review PR is waiting on a reviewer labels Jul 22, 2026
Only docker rm the shared reuse container after inspect shows a removable
state, and serialize recovery with a cross-process lock so transient
start flakes cannot yank a healthy instance from a peer nextest worker.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

S-waiting-on-author PR is waiting on author response

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Elasticsearch sink integration test flakes when connectors health never becomes ready

2 participants