[New Integration] Add Datadog Audit Logs integration

[hnguyen-coreweave]

Proposed commit message

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.
• I have verified that any added dashboard complies with Kibana's Dashboard good practices

Author's Checklist

• [ ]

How to test this PR locally

• yes, the package was deployed and have been running on an onprem cluster

Related issues

Screenshots

[elastic-vault-github-plugin-prod]

Reviewers

Buildkite won't run for external contributors automatically; you need to add a comment:

• /test : will kick off a build in Buildkite.

NOTE: https://github.com/elastic/integrations/blob/main/.buildkite/pull-requests.json contains all those details.

[hnguyen-coreweave]

@efd6 and @andrewkroh, I heard you guys are the go tos for reviewers

[hnguyen-coreweave]

/test

[qcorporation]

@vera-review-bot review

[qcorporation]

/test

[elastic-vault-github-plugin-prod]

Changelog link mismatch — expected https://github.com/elastic/integrations/pull/20526 in the following file(s):

• packages/datadog/changelog.yml

Tip

If expected, add the changelog-link-check:skip label to skip this check. Or, if an issue link was intended, use .../issues/<n> instead.

View Buildkite build
@hnguyen-coreweave

[infra-vault-gh-plugin-prod]

💔 Build Failed

• Buildkite Build
• Commit: fa5c04a

Failed CI Steps

• 📜 Check changelog PR links
• Check integrations datadog

History

[vera-review-bot]

Severity: 🟡 Medium confidence: medium path: packages/datadog/data_stream/audit/fields/fields.yml:95

datadog.metadata copies the entire vendor metadata object into a flattened field while ~30 of its leaves are already lifted into typed datadog.* fields, so most metadata content is indexed twice.

Details

default.yml lines 431-585 extract workflow_id, workflow_name, instance_id, step_name, bundle_id, request_id, track, session_id, agent_metadata., host_metadata., oauth_client., api_key.id, application_key.id, client., tool.name, run_as_user.*, usage.duration and more into dedicated fields; line 586-589 then copies the whole _dd.attributes.metadata object into datadog.metadata. Every one of those values is stored and indexed a second time under the flattened field, which is the same storage cost the deprecated preserve_duplicate_custom_fields pattern was removed for. Confirmed in the expected output: usage.duration appears at both line 6792 (inside datadog.metadata) and line 6869 (as datadog.usage.duration).

Recommendation:

Remove the keys that already have typed homes before copying the remainder, so the flattened field only carries genuinely unmodelled metadata:

- remove:
    field:
      - _dd.attributes.metadata.workflow_id
      - _dd.attributes.metadata.workflow_name
      - _dd.attributes.metadata.instance_id
      - _dd.attributes.metadata.usage
      - _dd.attributes.metadata.oauth_client
      - _dd.attributes.metadata.run_as_user
    ignore_missing: true
    tag: remove_promoted_metadata_keys
- set:
    field: datadog.metadata
    copy_from: _dd.attributes.metadata
    ignore_empty_value: true
    tag: set_metadata_remainder

---

🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills

⚠️ Automated review — verify suggestions before applying.

[hnguyen-coreweave]

I prefer to the keep the data in the original shape to avoid confusion for the readers in the future

[vera-review-bot]

Review summary

Issues found across the latest commits fa5c04a — 7 high, 10 medium, 2 low

• 🟠 Every dashboard panel references index-pattern id 'f33bf2a0-f206-4f43-9fa5-5aad898ea57c', a private data view from the author's Kibana. That saved object does not ship with the package, so all four panels break on install (link) (Unresolved)
• 🟠 conditions.kibana.version is "^8.0.0", which resolves to >=8.0.0 <9.0.0 and makes the package uninstallable on every 9.x stack (link) (Unresolved)
• 🟠 The changelog entry links to pull/20525 but this is PR 20526 (link) (Unresolved)
• 🟠 Polling mode emits interval:, which the aws-s3 input does not define, so the configured Polling Interval is silently ignored and the bucket is re-listed at the 120s default (link) (Unresolved)
• 🟠 packages/datadog/sample_event.json is a byte-for-byte copy of the pipeline test input (a {"events": [{"message": ...}]} wrapper of 24 raw log lines) placed at the package root (link) (Unresolved)
• 🟠 The user-supplied processors var is nested inside {{#if preserve_original_event}}, so it is silently discarded whenever preserve_original_event is false, and the bare processors: key is emitted even when nothing follows it. (link) (Unresolved)
• 🟠 The data stream has no fields/beats.yml, so the fields the aws-s3 input adds to every document (log.offset, input.type, aws.s3.bucket.name, aws.s3.object.key) are undeclared and fall to dynamic mapping. (link) (Unresolved)
• 🟡 owner.type is "community" while owner.github is the Elastic team elastic/integration-experience (link) (Unresolved)
• 🟡 The workflow_createdAt date processor can never succeed — Datadog sends Go's default time layout ("2026-05-21 19:32:51.72985 +0000 +0000"), which strict_date_optional_time cannot parse — and ignore_failure: true hides it, so datadog.workflow.created_at is never populated. (link) (Unresolved)
• 🟡 The dashboard has no data_stream.dataset filter and every panel query is empty, so all four visualisations aggregate over the whole logs-* data view instead of only datadog.audit. (link) (Unresolved)
• 🟡 The agent-side copy_fields that populates event.original omits fail_on_error: false and ignore_missing: true, so any event without a message field drops out of the pipeline instead of being forwarded. (link) (Unresolved)
• 🟡 The pipeline-level on_failure block is missing the third standard step that appends the preserve_original_event tag, so documents that fail processing are stripped of event.original by the Fleet final pipeline and become undebuggable. (link) (Unresolved)
• 🟡 The opening json processor uses ignore_failure: true, so a non-JSON line from the archive produces a near-empty document with no error signal (link) (Unresolved)
• 🟡 max_bytes is emitted at the top level but file_selectors is always rendered, and the aws-s3 input ignores the top-level reader config once selectors exist — so the Max File Size setting has no effect and the 10MB default applies. (link) (Unresolved)
• 🟡 The docs template contains no {{fields "audit"}} or {{event "audit"}} directives, so the built README ships without the exported-fields table and sample event that every integration's docs are expected to carry. (link) (Unresolved)
• 🟡 datadog.metadata copies the entire vendor metadata object into a flattened field while ~30 of its leaves are already lifted into typed datadog.* fields, so most metadata content is indexed twice. (link) (Unresolved)
• 🔵 Most of the ~110 processors in default.yml have no tag, so on_failure messages and the _ingest.on_failure_processor_tag placeholder cannot identify which processor failed (link) (Unresolved)
• 🔵 All four dashboard panels have an empty title ("title": ""), so the dashboard renders as four unlabelled charts (link) (Unresolved)
• 🟡 Changelog links a different PR number (link) (Unresolved)

Since this is a community PR, a new commit triggers another review — at most once every 30 minutes. I skip the PR while it's approved or has merge conflicts.

🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills

⚠️ Automated review — verify suggestions before applying.

[hnguyen-coreweave]

@qcorporation mind running the /test again? I believe I addressed most/all of the comments from the first review