[Stack 9.5.0-SNAPSHOT] Too many packages failing in daily job

[elastic-vault-github-plugin-prod]

• Stack version: 9.5.0-SNAPSHOT
• Packages:
  • admin_by_request_epm
  • apache_input_otel
  • atlassian_cloud
  • atlassian_confluence
  • atlassian_jira
  • azure_blob_storage
  • azure_network_watcher_nsg
  • azure_network_watcher_vnet
  • cisco_asa
  • cisco_secure_endpoint
  • claude_code
  • claude_cowork
  • docker_input_otel
  • elasticapm_input_otel
  • elasticsearch
  • endace
  • entityanalytics_ad
  • filelog_otel
  • forcepoint_web
  • hostmetrics_input_otel
  • httpcheck_otel
  • iis_input_otel
  • istio
  • jaeger_input_otel
  • kafka_input_otel
  • kubeletstats_input_otel
  • kubernetes
  • mongodb
  • mysql_input_otel
  • network_traffic
  • nginx_input_otel
  • okta
  • oracle
  • otlp_input_otel
  • prometheus_input_otel
  • prometheus_input_otel_raw
  • rapid7_insightvm
  • redis_input_otel
  • sql_server_input_otel
  • statsd_input_otel
  • system
  • ti_opencti
  • vectra_rux
  • verifier_otel
  • zipkin_input_otel
• Owners:
  • @elastic/ecosystem

Latest issue closed for the same test: #18575

First build failed: https://buildkite.com/elastic/integrations/builds/42293

Latest 5 failed builds:

• https://buildkite.com/elastic/integrations/builds/45768
• https://buildkite.com/elastic/integrations/builds/45842
• https://buildkite.com/elastic/integrations/builds/45912
• https://buildkite.com/elastic/integrations/builds/45973
• https://buildkite.com/elastic/integrations/builds/46037

[mrodm]

There were some packages failing due to network glitches:

• activemq
• journald
• iptables
• cassandra
• f5_bigip
• trellix_edr_cloud
• symnatec_endpoint_security
• cisco_umbrella
• aws (test related to apt-get failures)
• cloudfare_logpush (test related to apt-get failures)

After retry those packages, those tests failed due to apt-get were running successfully or at least without failures related to temporal network failures.

[mrodm]

Latest build has introduced a lot of errors for could not find the expected hits:
https://buildkite.com/elastic/integrations/builds/42520

[teresaromero]

After some investigation i got the following output for the majority of the packages. They might be failing because a cel error related to the otel collector. Assisted by claude

Summary

elastic/beats#48909 (merged 2026-02-18) was supposed to fix request-tracer path validation under the managed agent for the cel, httpjson, http_endpoint, and entityanalytics inputs. The fix is present on main and on the 9.4 release branch (the active branch when the PR landed). However the bug reproduces deterministically on stack 9.5.0-SNAPSHOT (verified locally 2026-05-08), because in 9.5 the elastic-agent runs filebeat as a receiver inside elastic-otel-collector (otelcol.component.id: filebeatreceiver/_agent-component/cel-default), and paths.Resolve(paths.Logs, input) in that runtime does not produce a path ending in the input name.

The integrations CI's daily build against 9.5.0-SNAPSHOT is currently flagging ~95 packages with the same "could not find expected hits" symptom (see elastic/integrations#18803). A static audit of elastic/integrations shows 75 of those 95 packages share this exact root cause, plus an additional 4 packages that match the audit signature but aren't yet in CI's failing list (latent / will start failing as soon as their tracer-enabled tests run on 9.5+).

Reproduction

Stack 9.5.0-SNAPSHOT, elastic-package v0.123.0:

elastic-package stack up -d -v --version=9.5.0-SNAPSHOT
cd packages/akamai
elastic-package test system -v --setup --config-file data_stream/siem/_dev/test/system/test-emulator-config.yml
docker logs $(docker ps --format '{{.Names}}' | grep agent-akamai-siem) 2>&1 | grep -i 'request tracer'

Observed: the cel-default component goes STARTING → FAILED with:

Permanent: Failed to start crawler: starting input failed: error while initializing input:
  request tracer path must be within "cel" path accessing 'filebeat.inputs.0.resource'

Input never executes; no HTTP requests reach the test emulator; ES logs-akamai.siem-* count stays at 0; system test asserts 144 hits and fails.

The compiled cel input config sent by Fleet (extracted from /api/fleet/agent_policies?full=true):

"resource": {
  "url": "http://akamai-siem-emulator:9903/siem/v1/configs/123456",
  "tracer": {
    "enabled": true,
    "filename": "../../logs/cel/http-request-trace-*.ndjson",
    "maxbackups": 5
  }
}

That ../../logs/cel/... relative pattern is emitted by every cel.yml.hbs / httpjson.yml.hbs / http_endpoint.yml.hbs template in elastic/integrations (473 data-stream templates across 146 packages — counts in the linked audit).

Why #48909 doesn't cover this

The error is generated at x-pack/filebeat/input/cel/config.go:327:

ok, err := httplog.IsPathInLogsFor(inputName, c.Tracer.Filename)
if err != nil { return err }
if !ok {
    return fmt.Errorf("request tracer path must be within %q path",
                       paths.Resolve(paths.Logs, inputName))
}

IsPathInLogsFor (in x-pack/filebeat/input/internal/httplog/roundtripper.go) uses paths.Resolve(paths.Logs, "cel") as the validation root. Under the legacy elastic-agent runtime, paths.Logs is /usr/share/elastic-agent/data/elastic-agent-XXX/logs and the join produces …/logs/cel, which is the same dir the tracer ultimately writes to — so the relative ../../logs/cel/foo resolves inside it and validation passes.

Under the OTel-collector receiver runtime in 9.5, the filebeat receiver is hosted by a different binary (/usr/share/elastic-agent/data/elastic-agent-XXX/components/elastic-otel-collector). paths.Logs in that process apparently does not point at the elastic-agent's logs/ tree, so the join produces a root that doesn't contain the resolved tracer path, and IsPathIn returns false.

Per #48909's body the same code path also runs when the tracer is disabled (tracerConfig.enabled() == nil || true is the default), which is consistent with the audit finding that several packages in #18803 have NO tests setting enable_request_tracer: true yet still fail.

[mrodm]

Thanks @teresaromero !

Buildkite build: https://buildkite.com/elastic/integrations/builds/42520 was running with the following stack:

╭──────────────────┬────────────────┬───────────────────┬───────────────────┬────────────╮
│     SERVICE      │    VERSION     │      STATUS       │ IMAGE BUILD DATE  │  VCS REF   │
├──────────────────┼────────────────┼───────────────────┼───────────────────┼────────────┤
│ elastic-agent    │ 9.5.0-SNAPSHOT │ running (healthy) │ 2026-05-07T03:05Z │ 7a962179d6 │
│ elasticsearch    │ 9.5.0-SNAPSHOT │ running (healthy) │ 2026-05-07T22:54Z │ 84580e8c70 │
│ fleet-server     │ 9.5.0-SNAPSHOT │ running (healthy) │ 2026-05-07T03:05Z │ 7a962179d6 │
│ kibana           │ 9.5.0-SNAPSHOT │ running (healthy) │ 2026-05-07T11:19Z │ 4eeb07e04b │
│ package-registry │ latest         │ running (healthy) │                   │            │
╰──────────────────┴────────────────┴───────────────────┴───────────────────┴────────────╯

Example of the error found for akamai:

{"log.level":"error","@timestamp":"2026-05-08T03:53:24.400Z","message":"error starting filebeat receiver{error 26 0  beat receiver run error: Failed to start crawler: starting input failed: error while initializing input: request tracer path must be within \"cel\" path accessing 'filebeat.inputs.0.resource'}","service.name":"filebeat","ecs.version":"1.6.0","log.origin":{"file.line":29,"file.name":"fbreceiver/receiver.go","function":"github.com/elastic/beats/v7/x-pack/filebeat/fbreceiver.(*filebeatReceiver).Start.func1"},"resource":{"service.instance.id":"8a494810-ec70-4971-a69a-58e9518ddc30","service.name":"/usr/share/elastic-agent/data/elastic-agent-7a9621/components/elastic-otel-collector","service.version":"9.5.0"},"otelcol.component.id":"filebeatreceiver/_agent-component/cel-default","otelcol.signal":"logs","otelcol.component.kind":"receiver","log":{"source":"cel-default"},"component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"cel-default","type":"cel"},"ecs.version":"1.6.0"}

Example of the error found for auth0:

{"log.level":"error","@timestamp":"2026-05-08T03:51:47.927Z","message":"error starting filebeat receiver{error 26 0  beat receiver run error: Failed to start crawler: starting input failed: error while initializing input: request tracer path must be within \"cel\" path accessing 'filebeat.inputs.0.resource'}","otelcol.signal":"logs","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"cel-default","type":"cel"},"service.name":"filebeat","resource":{"service.instance.id":"773c0bf1-6a9b-4c02-b124-ffb067d7b7aa","service.name":"/usr/share/elastic-agent/data/elastic-agent-7a9621/components/elastic-otel-collector","service.version":"9.5.0"},"otelcol.component.id":"filebeatreceiver/_agent-component/cel-default","otelcol.component.kind":"receiver","log":{"source":"cel-default"},"ecs.version":"1.6.0","log.origin":{"file.line":29,"file.name":"fbreceiver/receiver.go","function":"github.com/elastic/beats/v7/x-pack/filebeat/fbreceiver.(*filebeatReceiver).Start.func1"},"ecs.version":"1.6.0"}

According to this, that elastic-agent commit: https://github.com/elastic/elastic-agent/commits/7a962179d6db1d402f0475f3db0a98d272b4d2d3

That snapshot version contains this PR that could be related since it looks they are using the otel runtime now elastic/elastic-agent#13991 (cc @leehinman )

As it was also mentioned by the previous comment, this looks similar to the issue that was fixed a few weeks ago here elastic/beats#48909

cc @elastic/ecosystem @cmacknz

[cmacknz]

As it was also mentioned by the previous comment, this looks similar to the issue that was fixed a few weeks ago here elastic/beats#48909

CC @efd6 the request tracer path must be within \"cel\" path accessing 'filebeat.inputs.0.resource' error is back, though I am confused about why we no longer see this bug in agentless when CEL is run as a receiver.

[efd6]

@cmacknz The root cause is that NewBeatForReceiver in x-pack/libbeat/cmd/instance/beat.go intentionally creates a separate *paths.Path instance for each filebeat receiver and stores it in b.Info.Paths, but does not update the global paths.Paths singleton (because multiple receivers can coexist in the same collector process with different paths). However, httplog.IsPathInLogsFor and ResolvePathInLogsFor use the global paths.Resolve(paths.Logs, input) for validation. Under the OTel receiver runtime, it seems that the global is never initialized, so paths.Logs is empty and paths.Resolve(paths.Logs, "cel") returns just "cel" — a bare relative path. The tracer filename's ../../logs/cel/... then resolves outside that root and validation fails.

PR #48909's fix still works for the process runtime (where the global is correctly set) and for disabled tracers (the early return skips validation). The 9.5 regression only fires when both: (a) the OTel receiver runtime is used, and (b) the tracer is enabled. In the integrations CI, every system test config sets enable_request_tracer: true, which is why ~75 packages fail.

Re: why agentless doesn't show it: it is likely latent there too, but no test or real user enables request tracing in agentless, so the !c.Tracer.enabled() guard returns early before the broken path validation is reached.

The fix would be: (elastic/beats#50581)

• Remove the IsPathInLogsFor check from all Validate() methods (these run during config unpacking and can't access the correct per-receiver paths).
• Change ResolvePathInLogsFor to accept a *paths.Path parameter instead of using the global.
• Update all runtime callers (cel, httpjson, http_endpoint, entityanalytics/okta, jamf, azuread) to pass ctx.Agent.Paths, the correct per-instance paths from beat.Info.

@mrodm Can we have the sha refs for failures included in the original report?

[mrodm]

@mrodm Can we have the sha refs for failures included in the original report?

@efd6 If you refer about the SHA of the docker images running in the stack, that information was added a few months ago (starting with elastic-package v0.117.0)) in the Buildkite output itself (linked in the description). This is shown as part of the elastic-package stack status command in the VCS REF column
https://buildkite.com/elastic/integrations/builds/42614#019e1520-730c-4bd7-8d0b-f0642f771d11/L745-L753

╭──────────────────┬────────────────┬───────────────────┬───────────────────┬────────────╮
│     SERVICE      │    VERSION     │      STATUS       │ IMAGE BUILD DATE  │  VCS REF   │
├──────────────────┼────────────────┼───────────────────┼───────────────────┼────────────┤
│ elastic-agent    │ 9.5.0-SNAPSHOT │ running (healthy) │ 2026-05-10T03:11Z │ eac42baa7d │
│ elasticsearch    │ 9.5.0-SNAPSHOT │ running (healthy) │ 2026-05-08T16:38Z │ 78c313d002 │
│ fleet-server     │ 9.5.0-SNAPSHOT │ running (healthy) │ 2026-05-10T03:11Z │ eac42baa7d │
│ kibana           │ 9.5.0-SNAPSHOT │ running (healthy) │ 2026-05-10T11:27Z │ 1dfe3071d2 │
│ package-registry │ latest         │ running (healthy) │                   │            │
╰──────────────────┴────────────────┴───────────────────┴───────────────────┴────────────╯