Skip to content

Chargeback 0.5.1: data stream type and namespace filters - #20477

Open
JohannesMahne wants to merge 7 commits into
wip-johannes-chargebackfrom
feat/chargeback-23-ds-filters
Open

Chargeback 0.5.1: data stream type and namespace filters#20477
JohannesMahne wants to merge 7 commits into
wip-johannes-chargebackfrom
feat/chargeback-23-ds-filters

Conversation

@JohannesMahne

Copy link
Copy Markdown

Summary

  • Parse usage datastream into ds_type (first segment) and ds_namespace (last segment), with other fallback for non-conforming names.
  • Add ES|QL multi-select controls on Usage & Cost Allocation for data stream type and namespace (shared-deployment FinOps).
  • Document namespace-based ownership; bump package to 0.5.1 (includes unmerged 0.5.0 ES|QL control work on this branch).
  • Resolves elastic/elasticsearch-chargeback#23.

Test plan

  • elastic-package build --skip-validation produces chargeback-0.5.1.zip
  • E2E on Kibana/ES 9.4.2 via STACK_VERSION=9.4.2 ./scripts/release_chargeback.sh --cleanup-first --replace-dashboard --skip-build (companion repo)
  • Lookup aggregations: ds_type ∈ {logs, metrics, other}, ds_namespace ∈ {team_a, team_b, other}
  • Review Usage dashboard controls render and filter panels with sentinel default

Companion zip PR: open in elastic/elasticsearch-chargeback after this lands (or in parallel from the same package SHA).

Made with Cursor

JohannesMahne and others added 6 commits June 5, 2026 11:02
Replace options list controls with chained variable controls for deployment group, deployment, data tier, and data stream, and wire all usage panels to MV_INTERSECTS filters.

Co-authored-by: Cursor <cursoragent@cursor.com>
Replace options list controls with chained variable controls for deployment group, deployment, cost category, and cost type, and wire all billing panels to MV_INTERSECTS filters.

Co-authored-by: Cursor <cursoragent@cursor.com>
Switch Billing and Usage ES|QL controls from MV_INTERSECTS to MV_CONTAINS
for equivalent OR filtering on scalar fields with a lower ES floor (9.2).

Co-authored-by: Cursor <cursoragent@cursor.com>
…d nav

Embed multi-select ES|QL controls with __chargeback_unfiltered__ sentinel
filters, add packaged [Chargeback] Navigation links to all dashboards, and
bump transform pipeline references for the 0.5.0 release on Kibana 9.4+.
Re-export Billing, Usage, and Configuration dashboards plus the shared
Navigation links object from Kibana after E2E validation so packaged JSON
matches the verified 0.5.0 install state.

Co-authored-by: Cursor <cursoragent@cursor.com>
Parse Fleet data stream names on the usage path and expose type/namespace
ES|QL controls on Usage & Cost Allocation for shared-deployment FinOps.
Resolves elastic/elasticsearch-chargeback#23.

Co-authored-by: Cursor <cursoragent@cursor.com>
@github-actions

github-actions Bot commented Aug 3, 2026

Copy link
Copy Markdown
Contributor

Elastic Docs Style Checker (Vale)

Summary: 2 warnings, 1 suggestion found

⚠️ Warnings (2): Fix when the suggestion improves clarity or correctness.
File Line Rule Message
packages/chargeback/_dev/build/docs/README.md 286 Elastic.MenuArrowsBold Use '→' to separate menu items, not '' or '='. Example: Select Manage index → Add lifecycle policy.
packages/chargeback/_dev/build/docs/README.md 286 Elastic.MenuArrows Use '→' to separate menu items, not '' or '='. Example: Select Manage index → Add lifecycle policy.
💡 Suggestions (1): Optional style improvements. Apply when helpful.
File Line Rule Message
packages/chargeback/_dev/build/docs/README.md 243 Elastic.Semicolons Use semicolons judiciously.

The Vale linter checks documentation changes against the Elastic Docs style guide. To use Vale locally or report issues, refer to Elastic style guide for Vale.

@mergify

mergify Bot commented Aug 3, 2026

Copy link
Copy Markdown
Contributor

Tick the box to add this pull request to the merge queue (same as @mergifyio queue).

  • Queue this pull request

@github-actions

This comment has been minimized.

@github-actions

github-actions Bot commented Aug 3, 2026

Copy link
Copy Markdown
Contributor

TL;DR

Buildkite is failing for two independent reasons: (1) chargeback lint rejects the newly added kibana/links asset (item [links] is not allowed), and (2) the dev-script unit tests still fail on brittle fixed-length short-SHA assertions. Immediate unblock: remove/replace the unsupported kibana/links asset usage (or move to a package-spec version that supports it) and make get_release_commit hash assertions deterministic.

Remediation

  • Chargeback package failure: remove packages/chargeback/kibana/links/chargeback-e97bc218-1acc-474f-b47c-58d09ce83daf.json (or convert this navigation to a supported Kibana asset type for format_version: 3.5.0), then rerun Check integrations chargeback.
  • Dev script test failure: in dev/scripts/get_release_commit.sh and/or .buildkite/scripts/run_dev_scripts_tests.sh, stop asserting variable-length default short SHAs as exact 10 chars (either force --abbrev=10 in output or assert prefix/equality against full SHA), then rerun :bash: Dev scripts unit tests.
Investigation details

Root Cause

  1. Check integrations chargeback fails because this PR adds a new links saved object under kibana/links, but package lint rejects links in that folder for the current package spec context.
    • Added file in PR diff: packages/chargeback/kibana/links/chargeback-e97bc218-1acc-474f-b47c-58d09ce83daf.json (type: "links", added at diff hunk +1..+59).
    • manifest.yml remains format_version: 3.5.0 (packages/chargeback/manifest.yml:1 in PR diff).
  2. :bash: Dev scripts unit tests fails due to test brittleness around short SHA length; expected 10 chars but got 11-char abbreviations in this repo state.

Evidence

  • Build: https://buildkite.com/elastic/integrations/builds/47049
  • Job/step: Check integrations chargeback
    • Log excerpt: /tmp/gh-aw/buildkite-logs/integrations-check-integrations-chargeback.txt:117
    • item [links] is not allowed in folder [.../packages/chargeback/kibana]
  • Job/step: :bash: Dev scripts unit tests
    • Log excerpts:
      • /tmp/gh-aw/buildkite-logs/integrations-bash-dev-scripts-unit-tests.txt:117 expected 43bb655db0, got 43bb655db05
      • /tmp/gh-aw/buildkite-logs/integrations-bash-dev-scripts-unit-tests.txt:119 expected 8b024204a8, got 8b024204a84
      • /tmp/gh-aw/buildkite-logs/integrations-bash-dev-scripts-unit-tests.txt:121 expected fd04de398f, got fd04de398f1

Verification

  • Not run (workflow is read-only; diagnosis based on Buildkite logs + PR diff).

Follow-up

  • The short-SHA issue was already noted by the previous detective run; the new actionable delta here is the kibana/links asset rejection in the chargeback package lint step.

What is this? | From workflow: PR Buildkite Detective

Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.

…_commit

Remove unsupported kibana/links asset (package-spec rejects it) and embed
inline links panels on each Chargeback dashboard. Force --short=10 in
get_release_commit.sh and matching Buildkite unit tests.

Co-authored-by: Cursor <cursoragent@cursor.com>
@JohannesMahne
JohannesMahne requested a review from a team as a code owner August 3, 2026 15:04
@infra-vault-gh-plugin-prod

Copy link
Copy Markdown

💚 Build Succeeded

History

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant