feat(paloalto): create cortex XSOAR collector (#301)

[mariot]

Proposed changes

Testing Instructions

1. Step-by-step how to test
2. Environment or config notes

Related issues

• Closes feat(palo-alto-xsoar): create the collector #301

Checklist

• I consider the submitted work as finished
• I tested the code for its functionality
• I wrote test cases for the relevant uses case
• I added/update the relevant documentation (either on github or on notion)
• Where necessary I refactored code to improve the overall quality
• For bug fix -> I implemented a test that covers the bug

Further comments

[codecov]

Codecov Report

❌ Patch coverage is 98.25137% with 16 lines in your changes missing coverage. Please review.
✅ All tests successful. No failed tests found.

Files with missing lines	Patch %	Lines
palo-alto-cortex-xsoar/src/collector/models.py	90.56%	5 Missing ⚠️
palo-alto-cortex-xsoar/src/models/incident.py	91.89%	3 Missing ⚠️
...lo-alto-cortex-xsoar/src/services/trace_service.py	95.58%	3 Missing ⚠️
...-cortex-xsoar/src/models/settings/config_loader.py	91.30%	2 Missing ⚠️
palo-alto-cortex-xsoar/src/services/converter.py	87.50%	2 Missing ⚠️
...lo-alto-cortex-xsoar/src/services/ioc_extractor.py	98.38%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #348      +/-   ##
==========================================
+ Coverage   62.13%   68.32%   +6.19%     
==========================================
  Files          98      123      +25     
  Lines        4418     5333     +915     
==========================================
+ Hits         2745     3644     +899     
- Misses       1673     1689      +16     

Files with missing lines	Coverage Δ
palo-alto-cortex-xsoar/src/__init__.py	100.00% <100.00%> (ø)
palo-alto-cortex-xsoar/src/collector/__init__.py	100.00% <100.00%> (ø)
palo-alto-cortex-xsoar/src/collector/collector.py	100.00% <100.00%> (ø)
palo-alto-cortex-xsoar/src/collector/exception.py	100.00% <100.00%> (ø)
...-cortex-xsoar/src/collector/expectation_manager.py	100.00% <100.00%> (ø)
...o-alto-cortex-xsoar/src/collector/trace_manager.py	100.00% <100.00%> (ø)
palo-alto-cortex-xsoar/src/models/__init__.py	100.00% <100.00%> (ø)
...alo-alto-cortex-xsoar/src/models/authentication.py	100.00% <100.00%> (ø)
...-alto-cortex-xsoar/src/models/settings/__init__.py	100.00% <100.00%> (ø)
...-cortex-xsoar/src/models/settings/base_settings.py	100.00% <100.00%> (ø)
... and 15 more

📢 Thoughts on this report? Let us know!

🚀 New features to boost your workflow:

• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[guzmud]

Unless I'm mistaken, the API rate-limit responses leading to a backoff and retry is missing from the codebase (US.3 Error Handling & Resilience, AC-3.1, part of chunk 5 Hardening & Delivery).

Edit: handled by including the HTTP error 429 in the retry/backoff system

[Kakudou]

Mostly QoL, so i will conduct the execution/usage tests of the collector while waiting for those small fixes ;)

[Copilot]

Pull request overview

Adds a new standalone Python collector (palo-alto-cortex-xsoar) to query Cortex XSOAR incidents, match them against OpenAEV expectations (detection/prevention), and report results + traces back to OpenAEV, including Docker packaging and CircleCI integration.

Changes:

• Introduces the Palo Alto Cortex XSOAR collector implementation (config loading, API client, alert fetching, expectation matching, trace creation/submission).
• Adds a full pytest suite (factories/fixtures + unit tests) for the new collector.
• Wires the collector into build/test/release assets (Dockerfiles, compose example, CircleCI steps, metadata, README).

Reviewed changes

Copilot reviewed 48 out of 51 changed files in this pull request and generated 10 comments.

Show a summary per file

File	Description
palo-alto-cortex-xsoar/tests/test_trace_service.py	Tests for trace creation logic
palo-alto-cortex-xsoar/tests/test_trace_manager_extra.py	Extra tests for trace submission paths
palo-alto-cortex-xsoar/tests/test_trace_builder.py	Tests for trace builder utilities
palo-alto-cortex-xsoar/tests/test_expectation_service.py	Tests for expectation processing service
palo-alto-cortex-xsoar/tests/test_expectation_manager_extra.py	Extra tests for expectation manager behaviors
palo-alto-cortex-xsoar/tests/test_converter_and_extractor.py	Tests for converter + signature extraction helpers
palo-alto-cortex-xsoar/tests/test_collector.py	Higher-level collector behavior tests
palo-alto-cortex-xsoar/tests/test_collector_extra.py	Additional collector error-path tests
palo-alto-cortex-xsoar/tests/test_client_api.py	Tests for XSOAR HTTP API client
palo-alto-cortex-xsoar/tests/test_authentication.py	Tests for auth header generation
palo-alto-cortex-xsoar/tests/test_alert_fetcher.py	Tests for alert fetching + pagination
palo-alto-cortex-xsoar/tests/factories.py	Factory-boy fixtures for expectations/incidents/alerts
palo-alto-cortex-xsoar/tests/conftest.py	Shared pytest fixtures/mocks
palo-alto-cortex-xsoar/tests/init.py	Test package marker
palo-alto-cortex-xsoar/src/services/utils/trace_builder.py	Builds trace dictionaries from incidents
palo-alto-cortex-xsoar/src/services/utils/signature_extractor.py	Extracts/groups expectation signatures
palo-alto-cortex-xsoar/src/services/utils/init.py	Exports service utilities
palo-alto-cortex-xsoar/src/services/trace_service.py	Converts match results into OpenAEV trace models
palo-alto-cortex-xsoar/src/services/expectation_service.py	Orchestrates fetch/match/result creation
palo-alto-cortex-xsoar/src/services/exception.py	Service-layer exception types
palo-alto-cortex-xsoar/src/services/converter.py	Converts incident data into OAEV matching format
palo-alto-cortex-xsoar/src/services/client_api.py	Requests session + retrying XSOAR API client
palo-alto-cortex-xsoar/src/services/alert_fetcher.py	Time-window search + pagination fetcher
palo-alto-cortex-xsoar/src/services/init.py	Service package marker
palo-alto-cortex-xsoar/src/models/settings/palo_alto_cortex_xsoar_configs.py	Pydantic settings for XSOAR config
palo-alto-cortex-xsoar/src/models/settings/config_loader.py	Loads config from env/yaml/dotenv and flattens to daemon config
palo-alto-cortex-xsoar/src/models/settings/collector_configs.py	Collector/OpenAEV base settings models
palo-alto-cortex-xsoar/src/models/settings/base_settings.py	Shared BaseSettings configuration
palo-alto-cortex-xsoar/src/models/settings/init.py	Exports settings models
palo-alto-cortex-xsoar/src/models/incident.py	Pydantic models for XSOAR incident payloads
palo-alto-cortex-xsoar/src/models/authentication.py	Standard/advanced authentication header builder
palo-alto-cortex-xsoar/src/models/init.py	Exports top-level models
palo-alto-cortex-xsoar/src/config.yml.sample	Sample YAML configuration
palo-alto-cortex-xsoar/src/collector/trace_manager.py	Submits traces (bulk + fallback)
palo-alto-cortex-xsoar/src/collector/models.py	Pydantic models for results/traces/summary
palo-alto-cortex-xsoar/src/collector/expectation_manager.py	Fetch/process/update expectations and trigger tracing
palo-alto-cortex-xsoar/src/collector/exception.py	Collector-layer exception types
palo-alto-cortex-xsoar/src/collector/collector.py	CollectorDaemon integration + processing loop
palo-alto-cortex-xsoar/src/collector/init.py	Exports Collector
palo-alto-cortex-xsoar/src/main.py	Module entrypoint
palo-alto-cortex-xsoar/src/init.py	Package init/export
palo-alto-cortex-xsoar/README.md	Collector documentation and run/config instructions
palo-alto-cortex-xsoar/pyproject.toml	Collector packaging + dependencies
palo-alto-cortex-xsoar/manifest-metadata.json	Marketplace/manifest metadata
palo-alto-cortex-xsoar/Dockerfile_ubi9	UBI9 container build
palo-alto-cortex-xsoar/Dockerfile	Alpine container build
palo-alto-cortex-xsoar/docker-compose.yml	Local compose example
palo-alto-cortex-xsoar/.python-version	Local Python version pin
palo-alto-cortex-xsoar/.gitignore	Collector-local ignores
.circleci/config.yml	CI updates for tests + docker builds/push

[Kakudou]

Review Finding

During my review, I noticed a misbehavior in the current XSOAR incident retrieval and alert matching logic.

The endpoint we use retrieves Incidents, named cases in the UI. Those incidents are containers filled with Alerts.

So far, we filter incidents using fromDate and toDate.

Currently, fromDate is computed with:

start_time = end_time - self.time_window

where end_time is the expectation signature end_date, and self.time_window comes from the time_window setting:

Time window for PaloAltoCortexXSOAR alert searches when no date signature are provided (ISO 8601 format).

This creates a default incident retrieval window of 1h, which can be far too broad and noisy.

Required Fix

Use the expectation signature dates directly when available:

fromDate = start_date
toDate = end_date

Fallback to:

fromDate = now() - self.time_window
toDate = now()

only when no date signature is provided at all.

Detection Logic Change

start_date and end_date must now be treated as a weak but valid signature.

IP and ProcessName are deterministic enough in theory, but not reliable enough in the current XSOAR context. We will think later, in a future collector iteration, about more suitable signatures.

For now, after retrieving incidents, we must inspect the alerts contained inside each incident and keep only alerts whose own timestamp is inside the expectation time window.

If at least one alert exists inside [start_date, end_date], then:

is_detected = True

even if no other signature matches.

Then is_prevented must be evaluated from the matched alert status only.

Alerts outside the expectation time window must be ignored.

Checklist

• Use expectation start_date / end_date for incident retrieval.
• Use settings.time_window only as fallback when no date signature exists.
• Filter alerts inside each incident by alert timestamp.
• Ignore alerts outside [start_date, end_date].
• Set is_detected = True when at least one alert exists in the time window.
• Derive is_prevented only from the matched alert status.
• Keep IP / ProcessName as useful signatures, but not mandatory in XSOAR for detection matching, if not present, fallback to solely start/end_date matching.