Document missing event.ingested field on OTel-native data streams

[jamesagarside]

What this documents

event.ingested is not populated on OTel-native data streams (logs-*.otel-*, metrics-*.otel-*), and I couldn't find this stated anywhere in the docs. It surfaces as a confusing partial failure in Elastic Security prebuilt detection rules that use event.ingested as their timestamp override.

This adds a section to the data streams comparison page covering:

• The behaviour — event.ingested is set by .fleet_final_pipeline-1, which Fleet attaches as index.final_pipeline only to the data streams it manages. logs-otel@template and metrics-otel@template compose no final_pipeline, so nothing sets the field.
• The symptom — rules such as External Alerts report partial failure with The following indices are missing the timestamp override field "event.ingested". The rule still runs and falls back to @timestamp, but loses the ingest-lag protection the override provides.
• A workaround for classic OTel log data streams — via the logs@custom ingest pipeline extension point, writing to attributes.event.ingested with field_access_pattern: flexible.
• A caveat that logs@custom does not apply to wired Streams, where the only route is a copy_from processing step on a child stream, plus an explicit field declaration.

I also note that metrics-otel@template composes metrics@tsdb-settings and has no default_pipeline, so there is no equivalent hook for OTel metrics data streams.

Basis

This is based on behaviour I observed and verified first-hand on an Elastic Cloud Serverless Security project, Elasticsearch/Kibana 9.6. I've deliberately not claimed scope beyond that. I cross-checked the mechanism against the template and pipeline definitions in elastic/elasticsearch main:

• logs@default-pipeline.json — two processors: set @timestamp, then the logs@custom hook. No event.ingested.
• logs@settings.json — sets index.default_pipeline only, no final_pipeline.
• logs-otel@template.yaml / metrics-otel@template.yaml — composed_of lists, neither pulling in a final_pipeline.
• ecs@mappings.json — ecs_date dynamic template matches *.ingested, hence no explicit mapping needed on classic OTel log data streams.

On the workaround, I confirmed on the live cluster that after applying it the detection rule returned to succeeded across multiple executions, that the field is queryable under the bare name event.ingested and reported as such by _field_caps, and that field_access_pattern: flexible is required to write the dotted path into the attributes passthrough object. For wired streams I confirmed that Streamlang's set rejects Mustache syntax in value, that manual_ingest_pipeline is rejected in wired streams, and that _field_caps stayed empty until the field was declared on the stream.

Related public issue, linked in the page: elastic/elasticsearch#100324.

Notes for reviewers

• I placed this on the data streams comparison page as it reads as a compatibility gap, but it may belong on Limitations instead — happy to move it.
• I deliberately did not add a row to the summary table, because I have not verified whether classic APM data streams get event.ingested.
• Please correct anything I've got wrong, particularly the wired Streams details and whether the logs@custom workaround is something Elastic wants to recommend versus just documenting the gap.

[github-actions]

Elastic Docs AI PR menu

Check the box to run an AI review for this pull request.

• Review docs changes (docs-review). Status: not started.

Powered by GitHub Agentic Workflows and docs-actions. For more information, reach out to the docs team.