sentinel_one_cloud_funnel: fix routing and add S3-compatible endpoint configuration #20467
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,10 @@ | ||
| [!exec:echo] skip 'Skipping test requiring absent echo command' | ||
|
|
||
| exec echo ${PACKAGE_NAME} | ||
| stdout '^sentinel_one_cloud_funnel$' | ||
|
|
||
| exec echo ${DATA_STREAM} | ||
| stdout '^event$' | ||
|
|
||
| exec echo ${CURRENT_VERSION} | ||
| stdout '^[0-9]+\.[0-9]+\.[0-9]+$' |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,97 @@ | ||
| # Verify that events are correctly routed from the event data stream to | ||
| # category-specific data streams via routing rules when rerouting is enabled. | ||
| # | ||
| # The test data in minio provides: | ||
| # - 2 dns events → sentinel_one_cloud_funnel.dns (rerouted) | ||
| # - 1 process event → sentinel_one_cloud_funnel.process (rerouted) | ||
| # - 1 file event → sentinel_one_cloud_funnel.event (retained, reroute_file not enabled) | ||
|
|
||
| [!external_stack] skip 'Skipping external stack test.' | ||
| [!exec:jq] skip 'Skipping test requiring absent jq command' | ||
|
|
||
| # Connect to the running stack. | ||
| use_stack -profile ${CONFIG_PROFILES}/${PROFILE} | ||
|
|
||
| # Install an agent. | ||
| install_agent -profile ${CONFIG_PROFILES}/${PROFILE} -network_name NETWORK_NAME | ||
|
|
||
| # Start the minio service with pre-loaded test data. | ||
| docker_up -profile ${CONFIG_PROFILES}/${PROFILE} -network ${NETWORK_NAME} s1cf-minio | ||
|
|
||
| # Add the package resources. | ||
| add_package -profile ${CONFIG_PROFILES}/${PROFILE} | ||
|
|
||
| # Add the data stream policy with S3 polling against minio. | ||
| add_package_policy -profile ${CONFIG_PROFILES}/${PROFILE} test_config.yaml EVENT_DS | ||
|
|
||
| # The file event should remain in the event data stream (reroute_file not enabled). | ||
| get_docs -profile ${CONFIG_PROFILES}/${PROFILE} -want 1 -confirm 30s -timeout 5m ${EVENT_DS} | ||
| cp stdout event_docs.json | ||
| exec jq '.hits.hits[0]._source.sentinel_one_cloud_funnel.event.category' event_docs.json | ||
| stdout 'file' | ||
|
|
||
| # DNS events should be routed to the dns data stream. | ||
| get_docs -profile ${CONFIG_PROFILES}/${PROFILE} -want 2 -confirm 15s -timeout 2m logs-sentinel_one_cloud_funnel.dns-* | ||
| cp stdout dns_docs.json | ||
| exec jq -c '[.hits.hits[]._source.sentinel_one_cloud_funnel.event.category] | unique' dns_docs.json | ||
| stdout '\["dns"\]' | ||
|
|
||
| # Process events should be routed to the process data stream. | ||
| get_docs -profile ${CONFIG_PROFILES}/${PROFILE} -want 1 -confirm 15s -timeout 2m logs-sentinel_one_cloud_funnel.process-* | ||
| cp stdout proc_docs.json | ||
| exec jq '.hits.hits[0]._source.sentinel_one_cloud_funnel.event.category' proc_docs.json | ||
| stdout 'process' | ||
|
|
||
| # Clean up. | ||
| remove_package_policy -profile ${CONFIG_PROFILES}/${PROFILE} ${EVENT_DS} | ||
| uninstall_agent -profile ${CONFIG_PROFILES}/${PROFILE} -timeout 1m | ||
| docker_down s1cf-minio | ||
|
|
||
| -- test_config.yaml -- | ||
| input: aws-s3 | ||
| vars: | ||
| collect_s3_logs: true | ||
| non_aws_bucket_name: test-bucket | ||
| endpoint: http://svc-s1cf-minio:9000 | ||
| access_key_id: minioadmin | ||
| secret_access_key: minioadmin | ||
| path_style: true | ||
| region: us-east-1 | ||
| reroute_dns: true | ||
| reroute_process: true | ||
| data_stream: | ||
| vars: | ||
| preserve_original_event: true | ||
| interval: 10s | ||
|
|
||
| -- s1cf-minio/docker-compose.yml -- | ||
| services: | ||
| s1cf-minio: | ||
| image: minio/minio:RELEASE.2024-06-13T22-53-53Z | ||
| environment: | ||
| MINIO_ROOT_USER: minioadmin | ||
| MINIO_ROOT_PASSWORD: minioadmin | ||
| entrypoint: /bin/sh | ||
| command: | ||
| - -c | ||
| - | | ||
| minio server /data & | ||
| MINIO_PID=$$! | ||
| until curl -sf http://localhost:9000/minio/health/live; do sleep 1; done | ||
| mc alias set local http://localhost:9000 minioadmin minioadmin | ||
| mc mb local/test-bucket | ||
| mc cp /testdata/events.ndjson local/test-bucket/s1/cloud_funnel/events.ndjson | ||
| wait $$MINIO_PID | ||
| healthcheck: | ||
| test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"] | ||
| interval: 5s | ||
| timeout: 3s | ||
| retries: 10 | ||
| volumes: | ||
| - ./testdata:/testdata:ro | ||
|
|
||
| -- s1cf-minio/testdata/events.ndjson -- | ||
| {"event.category":"dns","event.time":1664811149495,"event.id":"TEST_DNS_001","event.type":"DNS Resolved","meta.event.name":"DNS","site.id":"123456789","account.id":"123456789","endpoint.name":"test-host","endpoint.os":"windows","endpoint.type":"laptop","agent.uuid":"test-agent-1","agent.version":"22.1.2.217","mgmt.id":"1337","mgmt.url":"test.sentinelone.org","os.name":"Windows10Pro","src.process.name":"chrome.exe","src.process.pid":1234,"src.process.uid":"AAAA0000","process.unique.key":"AAAA0000","trace.id":"TEST_DNS_001","packet.id":"PKT001","event.dns.request":"example.com","event.dns.response":"93.184.216.34","dataSource.name":"SentinelOne","dataSource.category":"security","i.scheme":"edr","i.version":"preprocess-lib-1.0","site.name":"TEST","group.id":"test-group","sca:atlantisIngestTime":1664811166298,"sca:ingestTime":1664811166,"timestamp":"18:32:29.495"} | ||
| {"event.category":"dns","event.time":1664811150000,"event.id":"TEST_DNS_002","event.type":"DNS Resolved","meta.event.name":"DNS","site.id":"123456789","account.id":"123456789","endpoint.name":"test-host","endpoint.os":"windows","endpoint.type":"laptop","agent.uuid":"test-agent-1","agent.version":"22.1.2.217","mgmt.id":"1337","mgmt.url":"test.sentinelone.org","os.name":"Windows10Pro","src.process.name":"firefox.exe","src.process.pid":5678,"src.process.uid":"BBBB0000","process.unique.key":"BBBB0000","trace.id":"TEST_DNS_002","packet.id":"PKT002","event.dns.request":"elastic.co","event.dns.response":"34.120.115.39","dataSource.name":"SentinelOne","dataSource.category":"security","i.scheme":"edr","i.version":"preprocess-lib-1.0","site.name":"TEST","group.id":"test-group","sca:atlantisIngestTime":1664811166298,"sca:ingestTime":1664811166,"timestamp":"18:32:30.000"} | ||
|
There was a problem hiding this comment. Severity: 🔵 Low The new routing test fixture uses real routable IP addresses and a real domain instead of documentation placeholders; replace them with RFC 5737 addresses and example.com. DetailsThe Recommendation: Swap the values for documentation-range equivalents; the routing assertions only depend on {"event.category":"dns","event.id":"TEST_DNS_001","event.dns.request":"blog.example.com","event.dns.response":"192.0.2.10"}
{"event.category":"dns","event.id":"TEST_DNS_002","event.dns.request":"infra-cdn.example.com","event.dns.response":"198.51.100.24"}🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills
|
||
| {"event.category":"process","event.time":1664811149470,"event.id":"TEST_PROC_001","event.type":"ProcessCreation","meta.event.name":"PROCESSCREATION","site.id":"123456789","account.id":"123456789","endpoint.name":"test-host","endpoint.os":"windows","endpoint.type":"server","agent.uuid":"test-agent-1","agent.version":"22.1.2.217","mgmt.id":"1337","mgmt.url":"test.sentinelone.org","os.name":"WindowsServer2019","src.process.name":"svchost.exe","src.process.pid":776,"src.process.uid":"CCCC0000","process.unique.key":"CCCC0000","trace.id":"TEST_PROC_001","packet.id":"PKT003","tgt.process.name":"notepad.exe","tgt.process.pid":9999,"tgt.process.uid":"DDDD0000","dataSource.name":"SentinelOne","dataSource.category":"security","i.scheme":"edr","i.version":"preprocess-lib-1.0","site.name":"TEST","group.id":"test-group","sca:atlantisIngestTime":1664811166298,"sca:ingestTime":1664811166,"timestamp":"18:32:29.470"} | ||
| {"event.category":"file","event.time":1664811149488,"event.id":"TEST_FILE_001","event.type":"FileCreation","meta.event.name":"FILECREATION","site.id":"123456789","account.id":"123456789","endpoint.name":"test-host","endpoint.os":"windows","endpoint.type":"desktop","agent.uuid":"test-agent-1","agent.version":"22.1.2.217","mgmt.id":"1337","mgmt.url":"test.sentinelone.org","os.name":"Windows10Pro","src.process.name":"explorer.exe","src.process.pid":2222,"src.process.uid":"EEEE0000","process.unique.key":"EEEE0000","trace.id":"TEST_FILE_001","packet.id":"PKT004","tgt.file.path":"C:\\Users\\test\\Documents\\test.txt","tgt.file.type":"UNKNOWN","dataSource.name":"SentinelOne","dataSource.category":"security","i.scheme":"edr","i.version":"preprocess-lib-1.0","site.name":"TEST","group.id":"test-group","sca:atlantisIngestTime":1664811166298,"sca:ingestTime":1664811166,"timestamp":"18:32:29.488"} | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,5 +1,7 @@ | ||
| title: Collect Event logs from SentinelOne Cloud Funnel. | ||
| elasticsearch: | ||
| dynamic_dataset: true | ||
|
There was a problem hiding this comment. Severity: 🟡 Medium Rerouted events now index successfully but every shipped dashboard filters on data_stream.dataset == sentinel_one_cloud_funnel.event, so routed documents are invisible; widen the dashboard filters to cover the routing targets. DetailsBefore this PR, a document rerouted to All six dashboards and the saved search in Recommendation: Replace the single phrase filter with one that also matches the routing targets, e.g. a {
"$state": { "store": "appState" },
"meta": {
"alias": "SentinelOne Cloud Funnel datasets",
"disabled": false,
"indexRefName": "kibanaSavedObjectMeta.searchSourceJSON.filter[0].meta.index",
"key": "data_stream.dataset",
"negate": false,
"params": ["sentinel_one_cloud_funnel.event", "sentinel_one_cloud_funnel.dns"],
"type": "phrases"
},
"query": {
"bool": {
"minimum_should_match": 1,
"should": [
{ "match_phrase": { "data_stream.dataset": "sentinel_one_cloud_funnel.event" } },
{ "match_phrase": { "data_stream.dataset": "sentinel_one_cloud_funnel.dns" } }
]
}
}
}If updating the dashboards is out of scope for this PR, please note the limitation in the README section that documents the rerouting options. 🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills
|
||
| dynamic_namespace: true | ||
| index_template: | ||
| settings: | ||
| analysis: | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,7 +1,7 @@ | ||
| format_version: "3.0.2" | ||
| name: sentinel_one_cloud_funnel | ||
| title: SentinelOne Cloud Funnel | ||
| version: "1.14.0" | ||
| description: Collect logs from SentinelOne Cloud Funnel with Elastic Agent. | ||
| type: integration | ||
| categories: | ||
|
|
@@ -65,14 +65,21 @@ | |
| multi: false | ||
| required: false | ||
| show_user: true | ||
| description: ARN of the AWS S3 bucket that will be polled for list operation. Required for collecting logs via the AWS S3 Bucket unless you set an Access Point ARN, or a Non-AWS Bucket Name for an S3-compatible service. In case both configurations are added, this one takes precedence. | ||
| - name: access_point_arn | ||
| type: text | ||
| title: "[S3] Access Point ARN" | ||
| multi: false | ||
| required: false | ||
| show_user: true | ||
| description: ARN of the AWS S3 Access Point that will be polled for list operation. Required for collecting logs via the AWS S3 Bucket unless you set a Bucket ARN, or a Non-AWS Bucket Name for an S3-compatible service. | ||
| - name: non_aws_bucket_name | ||
| type: text | ||
| title: "[S3] Non-AWS Bucket Name" | ||
| multi: false | ||
| required: false | ||
| show_user: false | ||
| description: Name of the S3 bucket for use with S3-compatible services (e.g. MinIO, Ceph). Use instead of Bucket ARN when not connecting to AWS S3. | ||
|
There was a problem hiding this comment. Severity: 🔵 Low The S3-compatible options (non_aws_bucket_name, endpoint, path_style, region) are all show_user: false and unmentioned in the README, so a MinIO/Ceph user only sees Bucket ARN and Access Point ARN; document the S3-compatible flow in _dev/build/docs/README.md. Details
There is also a hard requirement that is easy to miss: the Filebeat aws-s3 docs state that Recommendation: Add a short subsection to ### Collect data from an S3-compatible service (MinIO, Ceph, ...)
Under **Advanced options**, instead of a Bucket ARN or Access Point ARN, set:
- **[S3] Non-AWS Bucket Name** — the bucket name on the S3-compatible service.
- **Endpoint** — the full service URI, including the scheme, for example `https://s3.example.com:9000`.
- **Region** — required whenever a Non-AWS Bucket Name is set.
- **Path Style** — enable this; S3-compatible services generally require path-style access.
Authentication must use **Access Key ID** and **Secret Access Key**; IAM roles and shared credential profiles are AWS-only.Optionally, also point the two visible bucket options at the alternative so the UI is self-consistent: - name: bucket_arn
type: text
title: "[S3] Bucket ARN"
multi: false
required: false
show_user: true
description: ARN of the AWS S3 bucket that will be polled for list operation. Required when collecting logs via the AWS S3 Bucket unless you set an Access Point ARN, or a Non-AWS Bucket Name for an S3-compatible service. In case both configurations are added, this one takes precedence.🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills
There was a problem hiding this comment. Severity: 🔵 Low Adding non_aws_bucket_name as a third bucket option makes the neighbouring bucket_arn and access_point_arn descriptions inaccurate (they still claim one of the two ARNs is mandatory); update those two descriptions to mention the Non-AWS Bucket Name alternative. Details
Both statements are now false for the S3-compatible flow this PR adds, and they are the descriptions a MinIO/Ceph user reads first because those two vars are Recommendation: Mention the third option in both descriptions: - name: bucket_arn
type: text
title: "[S3] Bucket ARN"
multi: false
required: false
show_user: true
description: ARN of the AWS S3 bucket that will be polled for list operation. Required for collecting logs via the AWS S3 Bucket unless you set an Access Point ARN, or a Non-AWS Bucket Name for an S3-compatible service. In case both configurations are added, this one takes precedence.
- name: access_point_arn
type: text
title: "[S3] Access Point ARN"
multi: false
required: false
show_user: true
description: ARN of the AWS S3 Access Point that will be polled for list operation. Required for collecting logs via the AWS S3 Bucket unless you set a Bucket ARN, or a Non-AWS Bucket Name for an S3-compatible service.🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills
|
||
| - name: queue_url | ||
| type: text | ||
| title: "[SQS] Queue URL" | ||
|
|
@@ -169,6 +176,30 @@ | |
| # yvgJ38BRsFOtkRuAGSf6ZUwTO8JJRRIFnpUzXflAnGivK9M13D5GEQMmIl6U9Pvk | ||
| # sxSmbIUfc2SGJGCJD4I= | ||
| # -----END CERTIFICATE----- | ||
| - name: endpoint | ||
| type: text | ||
| title: Endpoint | ||
| multi: false | ||
| required: false | ||
| show_user: false | ||
| default: "" | ||
| description: URL of the entry point for an AWS web service. If set, the input will use this endpoint instead of the default AWS endpoint. | ||
|
There was a problem hiding this comment. Severity: 🔵 Low The Endpoint variable description shown in Fleet does not say the value must be a full URI including the scheme, unlike the README; restate the requirement in the manifest description. DetailsThe Filebeat Recommendation: Make the in-product description match the README and the Filebeat requirement: - name: endpoint
type: text
title: Endpoint
multi: false
required: false
show_user: false
default: ""
description: Full URI of the S3 service endpoint, including the scheme, for example `https://s3.example.com:9000`. Required when using a Non-AWS Bucket Name for an S3-compatible service. Leave empty to use the default AWS endpoint.🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills
|
||
| - name: path_style | ||
| type: bool | ||
| title: Path Style | ||
| multi: false | ||
| required: false | ||
| show_user: false | ||
| default: false | ||
| description: Enable path-style S3 access. Required when using non-AWS S3-compatible services. | ||
| - name: region | ||
| type: text | ||
| title: Region | ||
| multi: false | ||
| required: false | ||
| show_user: false | ||
| default: "" | ||
| description: AWS region for the S3 bucket. Required when using non_aws_bucket_name. | ||
| # Rerouting options | ||
| - name: reroute_command_script | ||
| type: bool | ||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Severity: 🟡 Medium
confidence: highpath: packages/sentinel_one_cloud_funnel/changelog.yml:4The 1.14.0 changelog does not mention that the logins data stream dataset was renamed from sentinel_one_cloud_funnel.login to .logins; add an entry for it since the backing index name changes for existing users.
Details
packages/sentinel_one_cloud_funnel/data_stream/logins/manifest.yml changes
datasetfromsentinel_one_cloud_funnel.logintosentinel_one_cloud_funnel.logins. That renames the index template and the data stream backing the logins stream (logs-sentinel_one_cloud_funnel.login-*->logs-sentinel_one_cloud_funnel.logins-*), which is user-visible on upgrade: any saved search, detection rule, or ES|QL query pinned to the old dataset stops matching, and previously ingested.logindocuments are no longer covered by a package-managed index template. The three entries added for 1.14.0 cover theevent.datasetconstant_keyword fix,dynamic_dataset/dynamic_namespace, and the new S3-compatible options, but none of them describes this rename, so nothing in the changelog tells an upgrading user their logins index name changed. This is also the kind of change the repo's breaking-change detector reports, so an explicit entry avoids a surprise later in CI.Recommendation:
Add a dedicated entry describing the rename (use
breaking-changeif the team agrees the index rename is breaking for existing installs):🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It never worked, so I'm OK with this.