feat(elastic): add Elastic Security collector (#423)

[SamuelHassine]

Summary

Adds an OpenAEV collector for Elastic Security. It validates OpenAEV detection expectations by querying Elasticsearch for Elastic Security detection alerts that match the attack signatures produced during a simulation, then reports a verdict and a trace linking back to the alerts.

Modeled on the existing splunk-es collector (pyoaev CollectorDaemon plus expectation/trace service providers), with the SIEM-specific layer rewritten for Elasticsearch.

What it does

• Queries the Elasticsearch _search API (default index .alerts-security.alerts-*)
• Authentication via API key (preferred) or HTTP basic
• Signature matching on ECS fields: source.ip, destination.ip, and parent process via url.path, bounded by the expectation time window
• Retry mechanism with a configurable offset to handle alert ingestion latency
• Trace generation with links to the Kibana Security alerts view
• Detection expectations only (Elastic is a detection source)

Implementation notes

• New Elasticsearch client building a _search query DSL, replacing the Splunk SPL client
• ECS-aware alert parsing (source.ip / destination.ip / url.path / rule name), tolerant of flattened or nested _source
• Auth validator: a configuration requires either an API key or both a username and a password
• Unit tests rewritten for the Elasticsearch query/response model

Review feedback addressed

• Removed the dead ElasticAlert._raw PrivateAttr: it was passed through the constructor (which Pydantic v2 ignores) so it was always None and never read.
• Corrected the _build_trace_url_from_expectation docstring to match the actual behavior: it builds a Kibana KQL query from source/destination IPs only, not the full Elasticsearch _search query.
• Fixed the README configuration section: the loader selects a single source (first of .env, config.yml, environment variables); it does not merge them.
• Fixed the documented default collector name (Elastic, not Elastic Security) to match ConfigLoaderCollector.
• The alpine Dockerfile keeps the [[ ... ]] override block to stay consistent with every other collector in the repo; the docker image build is green in CI.

Test plan

• bash run_test.sh elastic (pytest) green locally (43 passed)
• black / isort / flake8 (--ignore=E,W) clean
• CI green

Notes

• The collector icon (src/img/elastic-logo.png) is a placeholder and should be replaced with the official Elastic Security logo before release.

Closes #423

[Copilot]

Pull request overview

Adds a new Elastic Security collector to the OpenAEV collectors monorepo, modeled after the existing splunk-es collector, to validate detection expectations by querying Elasticsearch (POST /<index>/_search) and reporting verdicts and Kibana traces.

Changes:

• Introduces an Elasticsearch client (ElasticClientAPI) that builds query DSL, supports API key/basic auth, and retries for ingestion latency.
• Adds expectation + trace service providers and a collector daemon wiring them into the existing generic expectation/trace manager patterns.
• Adds pytest-based unit tests and polyfactory fixtures for the Elastic query/response model.

Reviewed changes

Copilot reviewed 46 out of 51 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
elastic/tests/test_create_collector.py	Validates collector initialization and config error cases.
elastic/tests/services/test_expectation_service_essential.py	Core expectation-service behavior tests (signatures, batching, matching).
elastic/tests/services/test_converter_essential.py	Converter unit tests for alert→OAEV mapping.
elastic/tests/services/test_client_api_essential.py	Client API tests for auth, query building, and retries.
elastic/tests/services/fixtures/factories.py	Polyfactory factories for configs, alerts, and test data.
elastic/tests/services/fixtures/init.py	Fixture package marker.
elastic/tests/services/conftest.py	Shared pytest fixtures for services tests.
elastic/tests/services/init.py	Services tests package marker.
elastic/tests/conftest.py	Test-wide mocking of OpenAEV calls and settings sources.
elastic/tests/init.py	Tests package marker.
elastic/src/services/utils/parent_process_parser.py	Extracts UUIDs and reconstructs URL/parent-process identifiers.
elastic/src/services/utils/config_loader.py	Wrapper config loader with logging and error handling.
elastic/src/services/utils/init.py	Exposes utils module API.
elastic/src/services/trace_service.py	Builds OpenAEV expectation traces (Kibana links).
elastic/src/services/models.py	Elastic alert/search response models and parsing helpers.
elastic/src/services/expectation_service.py	Expectation processing: signature extraction, fetch, convert, match.
elastic/src/services/exception.py	Elastic service exception hierarchy.
elastic/src/services/converter.py	Converts ElasticAlert objects into OAEV matchable structures.
elastic/src/services/client_api.py	Elasticsearch _search client with auth + retry + query DSL building.
elastic/src/services/init.py	Re-exports service components for external use.
elastic/src/py.typed	Marks package as typed for type checkers.
elastic/src/models/configs/elastic_configs.py	Pydantic settings for Elastic connectivity/auth/retry configuration.
elastic/src/models/configs/config_loader.py	Root settings loader and daemon config-flattening.
elastic/src/models/configs/collector_configs.py	Base collector/OpenAEV config settings models.
elastic/src/models/configs/base_settings.py	Shared BaseSettings configuration (env parsing, freezing, etc.).
elastic/src/models/configs/init.py	Config models export surface.
elastic/src/models/init.py	Exposes top-level ConfigLoader.
elastic/src/config.yml.sample	Sample YAML config for local/manual deployment.
elastic/src/collector/trace_service_provider.py	Protocol for trace services.
elastic/src/collector/trace_manager.py	Trace creation/submission logic to OpenAEV.
elastic/src/collector/signature_registry.py	Signature subscription/handler registry used by the collector.
elastic/src/collector/models.py	Collector-side Pydantic models (ExpectationResult/Trace/etc.).
elastic/src/collector/expectation_service_provider.py	Protocol for expectation service providers.
elastic/src/collector/expectation_manager.py	Generic expectation fetch/process/update loop.
elastic/src/collector/expectation_handler.py	Delegates expectation processing to the service provider.
elastic/src/collector/exception.py	Collector exception hierarchy.
elastic/src/collector/collector.py	CollectorDaemon implementation wiring services + managers.
elastic/src/collector/init.py	Collector package export surface.
elastic/src/.env.sample	Sample environment configuration.
elastic/src/main.py	CLI entrypoint to run the collector.
elastic/src/init.py	Package export surface.
elastic/README.md	Collector documentation and configuration reference.
elastic/pyproject.toml	Poetry packaging, dependencies, scripts, and tooling config.
elastic/manifest-metadata.json	Collector manifest metadata for platform/manager.
elastic/Dockerfile_ubi9	UBI9-based container build.
elastic/Dockerfile	Alpine-based container build.
elastic/docker-compose.yml	Local docker-compose example configuration.
elastic/.gitignore	Ignores config/build/cache artifacts.
elastic/.dockerignore	Excludes config/build/cache artifacts from image context.
elastic/.build.env	Build-time metadata for collector command.

[codecov]

Codecov Report

❌ Patch coverage is 85.80599% with 199 lines in your changes missing coverage. Please review.
✅ All tests successful. No failed tests found.

Files with missing lines	Patch %	Lines
elastic/src/collector/expectation_manager.py	71.92%	57 Missing ⚠️
elastic/src/services/expectation_service.py	84.03%	38 Missing ⚠️
elastic/src/collector/collector.py	48.14%	28 Missing ⚠️
elastic/src/services/trace_service.py	81.89%	21 Missing ⚠️
...lastic/src/services/utils/parent_process_parser.py	82.05%	14 Missing ⚠️
elastic/src/services/client_api.py	91.61%	13 Missing ⚠️
elastic/src/services/converter.py	89.10%	11 Missing ⚠️
elastic/src/models/configs/config_loader.py	69.56%	7 Missing ⚠️
elastic/src/collector/expectation_handler.py	95.23%	3 Missing ⚠️
elastic/src/services/utils/config_loader.py	88.46%	3 Missing ⚠️
... and 2 more

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #424      +/-   ##
==========================================
+ Coverage   61.84%   67.65%   +5.80%     
==========================================
  Files          98      125      +27     
  Lines        4385     5787    +1402     
==========================================
+ Hits         2712     3915    +1203     
- Misses       1673     1872     +199     

Files with missing lines	Coverage Δ
elastic/src/__init__.py	100.00% <100.00%> (ø)
elastic/src/collector/__init__.py	100.00% <100.00%> (ø)
elastic/src/collector/exception.py	100.00% <100.00%> (ø)
...stic/src/collector/expectation_service_provider.py	100.00% <100.00%> (ø)
elastic/src/collector/models.py	100.00% <100.00%> (ø)
elastic/src/collector/signature_registry.py	100.00% <100.00%> (ø)
elastic/src/collector/trace_service_provider.py	100.00% <100.00%> (ø)
elastic/src/models/__init__.py	100.00% <100.00%> (ø)
elastic/src/models/configs/__init__.py	100.00% <100.00%> (ø)
elastic/src/models/configs/base_settings.py	100.00% <100.00%> (ø)
... and 17 more

📢 Thoughts on this report? Let us know!

🚀 New features to boost your workflow:

• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[SamuelHassine]

Review and fix summary

Did an independent senior review of the full changed files (not just the diff) and addressed the Copilot review threads. Changes pushed in e34e6a9:

• Removed dead ElasticAlert._raw PrivateAttr (src/services/models.py): it was assigned through the constructor, which Pydantic v2 ignores, so it was always None and never read. Dropped the field, the constructor argument and the unused PrivateAttr import.
• Corrected _build_trace_url_from_expectation docstring (src/services/trace_service.py): it builds a Kibana KQL query from source/destination IPs only, not the full Elasticsearch _search query (no url.path match, no @timestamp window).
• Fixed README configuration precedence: the loader selects a single source (first of .env, config.yml, environment variables) and does not merge them.
• Fixed the documented default collector name (Elastic, not Elastic Security) to match ConfigLoaderCollector.

Deliberately not changed:

• The alpine Dockerfile keeps the [[ ... ]] override block. Every other collector in the repo uses the same form and the build_docker_images job is green; switching to POSIX [ ... ] should be a single repo-wide change, not a per-collector divergence.

Status:

• All CI checks are green (tests, linter, formatting, signed commits, codecov patch/project, docker image build).
• All 4 review threads replied to and resolved.
• mergeable: MERGEABLE. The only thing left is a maintainer approval (REVIEW_REQUIRED); I could not approve since I am the PR author.