Skip to content

Crowdstrike: use inclusive >= cursor boundary for host data stream #20524

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
chemamartinez merged 3 commits into from
Aug 5, 2026
Merged

Crowdstrike: use inclusive >= cursor boundary for host data stream #20524

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
2a2df62
Use inclusive >= cursor boundary for host data stream
chemamartinez Aug 4, 2026
4fa9876
Update changelog
chemamartinez Aug 4, 2026
a7774fc
Reset offset on failures
chemamartinez Aug 5, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions packages/crowdstrike/changelog.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,9 @@
# newer versions go on top
- version: "4.5.1"
changes:
- description: Use inclusive >= cursor boundary in host CEL program to avoid permanently skipping same-timestamp records at page or error boundaries.
type: bugfix
link: https://github.com/elastic/integrations/pull/20524
- version: "4.5.0"
changes:
- description: Map CommandHistory events to ECS process fields.
Expand Down
10 changes: 8 additions & 2 deletions packages/crowdstrike/data_stream/host/_dev/test/policy/test-default.expected
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -45,7 +45,10 @@ inputs:
optional.of(
[
[
?filter.optMap(f, "modified_timestamp:>\"" + f + "\""),
// Inclusive lower bound: records sharing the boundary timestamp can span a page
// or error boundary, so ">=" avoids permanently skipping same-timestamp records.
// Re-fetched records are de-duplicated downstream by the fingerprint _id
?filter.optMap(f, "modified_timestamp:>=\"" + f + "\""),
?state.?query.optMap(q, "(" + q + ")"),
].join("+"),
]
Expand Down Expand Up @@ -154,7 +157,10 @@ inputs:
"sort": ["modified_timestamp|asc"],
"filter": [
[
"modified_timestamp:>'" + start_time + "'",
// Inclusive lower bound: records sharing the boundary timestamp can span a page
// or error boundary, so ">=" avoids permanently skipping same-timestamp records.
// Re-fetched records are de-duplicated downstream by the fingerprint _id
"modified_timestamp:>='" + start_time + "'",
?state.?query.optMap(q, "(" + q + ")"),
].join("+"),
],
Expand Down
10 changes: 8 additions & 2 deletions packages/crowdstrike/data_stream/host/agent/stream/cel.yml.hbs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -64,7 +64,10 @@ program: |-
optional.of(
[
[
?filter.optMap(f, "modified_timestamp:>\"" + f + "\""),
// Inclusive lower bound: records sharing the boundary timestamp can span a page
// or error boundary, so ">=" avoids permanently skipping same-timestamp records.
// Re-fetched records are de-duplicated downstream by the fingerprint _id
?filter.optMap(f, "modified_timestamp:>=\"" + f + "\""),

@vera-review-bot vera-review-bot Bot Aug 4, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Severity: 🟠 High confidence: medium path: packages/crowdstrike/data_stream/host/agent/stream/cel.yml.hbs:70

The inclusive >= boundary fixes timestamp skipping, but the GovCloud branch still permanently skips records at an error boundary because state.offset is never reset on the error paths; add "offset": 0 to both GovCloud error returns.

Details

This change makes the timestamp lower bound inclusive so same-timestamp records are not skipped across page/error boundaries. In the GovCloud branch that fix is incomplete, because pagination here is offset-based and state.offset survives the error paths.

state.with() merges keys, and neither GovCloud error return sets offset, so the previous value is preserved:

  • GET error (lines 93-108) returns {events, want_more: false, next: {}} - offset keeps whatever the previous execution left.
  • POST error (lines 140-155) is reached after the GET success block already advanced offset to int(state.offset) + body.resources.size() (line 87), so offset is left pointing past the records that were never published.

offset is only reset to 0 on the success path when want_more is false. On the next execution want_more is false, so the filter is rebuilt from state.cursor.last_timestamp (or, if the object-shaped error caused the agent to drop the cursor, from now - initial_interval) - a fresh, much narrower result set - while offset is still N. The API then skips the first N records of that new result set, and nothing ever comes back for them. A single transient 5xx on the entities POST during the first page therefore drops that entire page permanently.

The commercial branch does not have this problem: its GET error return sets next: {}, which clears next.page_token and restarts pagination cleanly. The GovCloud branch should reset its pagination state the same way.

Recommendation:

Reset offset to 0 in both GovCloud error returns so a failed execution restarts pagination from the top of the freshly filtered result set:

{
  "events": {
    "error": {
      "code": string(get_resp.StatusCode),
      "id": string(get_resp.Status),
      "message": "GET: " + (
        (size(get_resp.Body) != 0) ?
          string(get_resp.Body)
        :
          string(get_resp.Status) + " (" + string(get_resp.StatusCode) + ")"
      ),
    },
  },
  "offset": 0,
  "want_more": false,
  "next": {},
}

and likewise for the entities POST error return:

{
  "events": {
    "error": {
      "code": string(post_resp.StatusCode),
      "id": string(post_resp.Status),
      "message": "POST: " + (
        (size(post_resp.Body) != 0) ?
          string(post_resp.Body)
        :
          string(post_resp.Status) + " (" + string(post_resp.StatusCode) + ")"
      ),
    },
  },
  "offset": 0,
  "want_more": false,
  "next": {},
}

Regenerate _dev/test/policy/test-default.expected after the change.

🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills

⚠️ Automated review — verify suggestions before applying.

@chemamartinez chemamartinez Aug 5, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems to be a valid concern.

?state.?query.optMap(q, "(" + q + ")"),
].join("+"),
]
Expand Down Expand Up @@ -173,7 +176,10 @@ program: |-
"sort": ["modified_timestamp|asc"],
"filter": [
[
"modified_timestamp:>'" + start_time + "'",
// Inclusive lower bound: records sharing the boundary timestamp can span a page
// or error boundary, so ">=" avoids permanently skipping same-timestamp records.
// Re-fetched records are de-duplicated downstream by the fingerprint _id
"modified_timestamp:>='" + start_time + "'",
?state.?query.optMap(q, "(" + q + ")"),
].join("+"),
],
Expand Down
2 changes: 1 addition & 1 deletion packages/crowdstrike/manifest.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
name: crowdstrike
title: CrowdStrike
version: "4.5.0"
version: "4.5.1"
description: Collect logs from Crowdstrike with Elastic Agent.
type: integration
format_version: "3.4.0"
Expand Down
Loading