Chargeback 0.5.1: data stream type and namespace filters

.buildkite/scripts/run_dev_scripts_tests.sh

@@ -60,7 +60,7 @@ setup_dummy_repo() {
     git -C "$tmpdir" add .
     git -C "$tmpdir" commit -q -m "Release flat_pkg 1.0.0"
     mkdir -p "${tmpdir}/.state"
-    git -C "$tmpdir" rev-parse --short HEAD > "${tmpdir}/.state/flat_pkg_1.0.0"
+    git -C "$tmpdir" rev-parse --short=10 HEAD > "${tmpdir}/.state/flat_pkg_1.0.0"
 
     printf 'name: flat_pkg\nversion: 1.1.0\n' > "${tmpdir}/packages/flat_pkg/manifest.yml"
     git -C "$tmpdir" add .
@@ -70,7 +70,7 @@ setup_dummy_repo() {
     printf 'name: nested_pkg\nversion: 2.0.0\n' > "${tmpdir}/packages/group/nested_pkg/manifest.yml"
     git -C "$tmpdir" add .
     git -C "$tmpdir" commit -q -m "Release nested_pkg 2.0.0"
-    git -C "$tmpdir" rev-parse --short HEAD > "${tmpdir}/.state/nested_pkg_2.0.0"
+    git -C "$tmpdir" rev-parse --short=10 HEAD > "${tmpdir}/.state/nested_pkg_2.0.0"
 
     printf 'name: nested_pkg\nversion: 2.1.0\n' > "${tmpdir}/packages/group/nested_pkg/manifest.yml"
     git -C "$tmpdir" add .

dev/scripts/get_release_commit.sh

@@ -61,10 +61,10 @@ COMMITS=()
 while IFS= read -r line; do
     [[ -n "$line" ]] && COMMITS+=("$line")
 done < <(
-    git log --oneline \
+    git log --format='%H' \
         -G "^version: ['\"]?${ESCAPED_VERSION}['\"]?[[:space:]]*$" \
         -- "$MANIFEST" \
-        | awk 'NF {print $1}' | sort -u
+        | sort -u
 )
 
 # Among matching commits, keep only those that *added* (not removed) the version.
@@ -82,4 +82,5 @@ if [[ -z "$FOUND" ]]; then
     exit 1
 fi
 
-echo "$FOUND"
+# Fixed-length short SHA so callers/tests are not brittle to git's dynamic --abbrev.
+git rev-parse --short=10 "$FOUND"

packages/chargeback/_dev/build/docs/README.md

@@ -22,6 +22,7 @@
 
 **Monitoring cluster:**
 - Must be on Elasticsearch version **9.2.0+** due to the use of smart [ES|QL LOOKUP JOIN](https://www.elastic.co/docs/reference/query-languages/esql/esql-lookup-join) (conditional joins) in transforms and dashboard queries.
+- Kibana version **9.4.0+** is required for package **0.5.0** and later. **Kibana 9.3 is not supported** — Billing and Usage dashboards fail to render (`No embeddable factory found for type: vis`) and lack GA support for ES|QL multi-select variable controls.
 - This is where the Chargeback integration should be installed.
 
 **Required integrations:**
@@ -148,9 +149,9 @@
 
 **Usage transforms** (from monitoring indices):
 5. **`cluster_deployment_contribution`**: indexing, querying, and storage metrics per deployment/day.
-6. **`cluster_datastream_contribution`**: same metrics split by data stream.
+6. **`cluster_datastream_contribution`**: same metrics split by data stream. The usage ingest pipeline also sets `ds_type` and `ds_namespace` from the data stream name.
 7. **`cluster_tier_contribution`**: same metrics split by data tier.
-8. **`cluster_tier_and_ds_contribution`**: same metrics split by both tier and data stream.
+8. **`cluster_tier_and_ds_contribution`**: same metrics split by both tier and data stream (includes `ds_type` and `ds_namespace`).
 
 These transforms produce lookup indices queried by the dashboards using ES|QL LOOKUP JOINs.
 
@@ -188,6 +189,8 @@
 
 **Source:** `cluster_tier_contribution_lookup` and related usage lookups. Totals reflect the chargeable pool (allocatable data-tier ECU discounted by utilization) and will not equal the full invoice. ML, Kibana, snapshots, and data transfer are excluded.
 
+**Controls:** deployment group, deployment name, data tier, full data stream name, **data stream type** (`ds_type`), and **data stream namespace** (`ds_namespace`).
+
 Sections:
 - **Deployment cost allocation (usage-based)**: normalized cost per deployment split by data tier (usage-weighted). Shows which deployments consume the most of their chargeable pool across tiers.
 - **Datatiers / utilization**: provisioned capacity vs chargeable pool, utilization p95 per deployment.
@@ -215,9 +218,35 @@
 
 When `node_stats` is missing for a deployment/day, utilization defaults to 100%.
 
+## Shared deployments and data stream namespaces
+
+Document-level chargeback (for example by a field such as `application.id`) is not available: billing and monitoring allocate at deployment, tier, and data stream level. On a **shared deployment**, assign each team a unique [Fleet data stream namespace](https://www.elastic.co/docs/reference/fleet/data-streams) so ownership follows the naming scheme:
+
+```text
+<type>-<dataset>-<namespace>
+```
+
+Examples: `logs-nginx.access-team_a` → `ds_type=logs`, `ds_namespace=team_a`.
+
+Chargeback parses every usage `datastream` value into:
+
+| Field | Rule |
+|-------|------|
+| `ds_type` | First `-`-separated segment |
+| `ds_namespace` | Last `-`-separated segment |
+| (not a v1 control) | Middle segment(s) are the dataset |
+
+Names without a hyphen set both fields to `other` so panels keep working. Use the **Data stream type** and **Data stream namespace** controls on **Usage & Cost Allocation** to filter cost. A dataset control is intentionally out of scope for this release (high cardinality; weak team signal). The full data stream name control remains available separately.
+
+Guidance:
+
+1. Give each team a unique namespace (no hyphens; max 100 bytes).
+2. Use separate agent policies or per-integration namespace overrides so data lands in distinct streams.
+3. Filter or break down by `ds_namespace` (and optionally `ds_type`) on the Usage dashboard.
+
 ## Deployment Groups
 
-The integration supports organising deployments into logical groups using the `chargeback_group` tag on ESS Billing deployments. This enables cost allocation and filtering by team, project, or any organisational structure.
+The integration supports organizing deployments into logical groups using the `chargeback_group` tag on ESS Billing deployments. This enables cost allocation and filtering by team, project, or any organizational structure.
 
 To assign a deployment to a chargeback group, add a tag in the Elastic Cloud console in the format:
 
@@ -243,6 +272,19 @@
 
 ## Upgrade Notes
 
+### Upgrading to 0.5.1
+
+1. Upgrade the Fleet package to **0.5.1** (Kibana **9.4.0+** required, same as 0.5.0).
+2. Usage transforms that write through the usage pipeline are reinstalled with `fleet_transform_version: 0.5.1` so lookup documents pick up `ds_type` and `ds_namespace`. Reset those transforms if existing lookup docs lack the new fields.
+3. The Usage & Cost Allocation dashboard gains **Data stream type** and **Data stream namespace** controls. Delete stale dashboard saved objects if duplicates appear after upgrade.
+
+### Upgrading to 0.5.0
+
+1. Upgrade the Fleet package to **0.5.0**.
+2. Upgrade Kibana to **9.4.0+** before installing 0.5.0. Kibana 9.3 cannot render the updated Billing and Usage dashboards.
+3. Transforms are reinstalled with `fleet_transform_version: 0.5.0` (pipeline logic unchanged). No lookup index recreation is required.
+4. The Billing and Usage dashboards are replaced with versions that use chained ES|QL variable controls instead of legacy options-list controls. Delete stale dashboard saved objects from **Stack Management > Saved Objects** if duplicates appear after upgrade.
+
 ### Upgrading to 0.4.0
 
 1. Upgrade the Fleet package to **0.4.0**.

packages/chargeback/changelog.yml

@@ -1,4 +1,14 @@
 # newer versions go on top
+- version: 0.5.1
+  changes:
+    - description: "Parse data stream names into ds_type (first segment) and ds_namespace (last segment) on the usage path, with other fallback for non-conforming names. Add ES|QL multi-select controls on the Usage & Cost Allocation dashboard to filter by data stream type and namespace for shared-deployment FinOps. Bump transform pipeline references and fleet_transform_version to 0.5.1."
+      type: enhancement
+      link: https://github.com/elastic/elasticsearch-chargeback/issues/23
+- version: 0.5.0
+  changes:
+    - description: "Replace legacy options-list dashboard controls with embedded ES|QL multi-select variable controls on the Billing Components Overview and Usage & Cost Allocation dashboards. Controls ship with a __chargeback_unfiltered__ sentinel default so fresh installs avoid empty multi-select param errors on Elasticsearch 9.4+. Panel queries use MV_INTERSECTS filters. Add a shared [Chargeback] Navigation links object across all three dashboards. Requires Kibana 9.4.0+. Bump transform pipeline references and fleet_transform_version to 0.5.0 (pipeline logic unchanged)."
+      type: enhancement
+      link: https://github.com/elastic/elasticsearch-chargeback/issues/8
 - version: 0.4.0
   changes:
     - description: "Split monolithic dashboard into three focused dashboards: '[Chargeback] Billing Components Overview' (full invoice by deployment group and billing component), '[Chargeback] Usage & Cost Allocation' (usage-based chargeable pool allocation by tier and data stream), and '[Chargeback] Configuration' (rate, weights, and date-window reference). Resolves elasticsearch-chargeback#8."

packages/chargeback/docs/README.md

@@ -22,6 +22,7 @@ Chargeback costs are presented based on a configured rate and unit, used to conv
 
 **Monitoring cluster:**
 - Must be on Elasticsearch version **9.2.0+** due to the use of smart [ES|QL LOOKUP JOIN](https://www.elastic.co/docs/reference/query-languages/esql/esql-lookup-join) (conditional joins) in transforms and dashboard queries.
+- Kibana version **9.4.0+** is required for package **0.5.0** and later. **Kibana 9.3 is not supported** — Billing and Usage dashboards fail to render (`No embeddable factory found for type: vis`) and lack GA support for ES|QL multi-select variable controls.
 - This is where the Chargeback integration should be installed.
 
 **Required integrations:**
@@ -148,9 +149,9 @@ The integration creates eight transforms to aggregate cost and usage data:
 
 **Usage transforms** (from monitoring indices):
 5. **`cluster_deployment_contribution`**: indexing, querying, and storage metrics per deployment/day.
-6. **`cluster_datastream_contribution`**: same metrics split by data stream.
+6. **`cluster_datastream_contribution`**: same metrics split by data stream. The usage ingest pipeline also sets `ds_type` and `ds_namespace` from the data stream name.
 7. **`cluster_tier_contribution`**: same metrics split by data tier.
-8. **`cluster_tier_and_ds_contribution`**: same metrics split by both tier and data stream.
+8. **`cluster_tier_and_ds_contribution`**: same metrics split by both tier and data stream (includes `ds_type` and `ds_namespace`).
 
 These transforms produce lookup indices queried by the dashboards using ES|QL LOOKUP JOINs.
 
@@ -188,6 +189,8 @@ Answers: *which data streams and tiers drive cost, and how efficiently are we us
 
 **Source:** `cluster_tier_contribution_lookup` and related usage lookups. Totals reflect the chargeable pool (allocatable data-tier ECU discounted by utilization) and will not equal the full invoice. ML, Kibana, snapshots, and data transfer are excluded.
 
+**Controls:** deployment group, deployment name, data tier, full data stream name, **data stream type** (`ds_type`), and **data stream namespace** (`ds_namespace`).
+
 Sections:
 - **Deployment cost allocation (usage-based)**: normalized cost per deployment split by data tier (usage-weighted). Shows which deployments consume the most of their chargeable pool across tiers.
 - **Datatiers / utilization**: provisioned capacity vs chargeable pool, utilization p95 per deployment.
@@ -215,9 +218,35 @@ Do not expect the chargeable pool to equal the full deployment bill. Non-allocat
 
 When `node_stats` is missing for a deployment/day, utilization defaults to 100%.
 
+## Shared deployments and data stream namespaces
+
+Document-level chargeback (for example by a field such as `application.id`) is not available: billing and monitoring allocate at deployment, tier, and data stream level. On a **shared deployment**, assign each team a unique [Fleet data stream namespace](https://www.elastic.co/docs/reference/fleet/data-streams) so ownership follows the naming scheme:
+
+```text
+<type>-<dataset>-<namespace>
+```
+
+Examples: `logs-nginx.access-team_a` → `ds_type=logs`, `ds_namespace=team_a`.
+
+Chargeback parses every usage `datastream` value into:
+
+| Field | Rule |
+|-------|------|
+| `ds_type` | First `-`-separated segment |
+| `ds_namespace` | Last `-`-separated segment |
+| (not a v1 control) | Middle segment(s) are the dataset |
+
+Names without a hyphen set both fields to `other` so panels keep working. Use the **Data stream type** and **Data stream namespace** controls on **Usage & Cost Allocation** to filter cost. A dataset control is intentionally out of scope for this release (high cardinality; weak team signal). The full data stream name control remains available separately.
+
+Guidance:
+
+1. Give each team a unique namespace (no hyphens; max 100 bytes).
+2. Use separate agent policies or per-integration namespace overrides so data lands in distinct streams.
+3. Filter or break down by `ds_namespace` (and optionally `ds_type`) on the Usage dashboard.
+
 ## Deployment Groups
 
-The integration supports organising deployments into logical groups using the `chargeback_group` tag on ESS Billing deployments. This enables cost allocation and filtering by team, project, or any organisational structure.
+The integration supports organizing deployments into logical groups using the `chargeback_group` tag on ESS Billing deployments. This enables cost allocation and filtering by team, project, or any organizational structure.
 
 To assign a deployment to a chargeback group, add a tag in the Elastic Cloud console in the format:
 
@@ -243,6 +272,19 @@ For more information, refer to the [Elastic documentation](https://www.elastic.c
 
 ## Upgrade Notes
 
+### Upgrading to 0.5.1
+
+1. Upgrade the Fleet package to **0.5.1** (Kibana **9.4.0+** required, same as 0.5.0).
+2. Usage transforms that write through the usage pipeline are reinstalled with `fleet_transform_version: 0.5.1` so lookup documents pick up `ds_type` and `ds_namespace`. Reset those transforms if existing lookup docs lack the new fields.
+3. The Usage & Cost Allocation dashboard gains **Data stream type** and **Data stream namespace** controls. Delete stale dashboard saved objects if duplicates appear after upgrade.
+
+### Upgrading to 0.5.0
+
+1. Upgrade the Fleet package to **0.5.0**.
+2. Upgrade Kibana to **9.4.0+** before installing 0.5.0. Kibana 9.3 cannot render the updated Billing and Usage dashboards.
+3. Transforms are reinstalled with `fleet_transform_version: 0.5.0` (pipeline logic unchanged). No lookup index recreation is required.
+4. The Billing and Usage dashboards are replaced with versions that use chained ES|QL variable controls instead of legacy options-list controls. Delete stale dashboard saved objects from **Stack Management > Saved Objects** if duplicates appear after upgrade.
+
 ### Upgrading to 0.4.0
 
 1. Upgrade the Fleet package to **0.4.0**.

packages/chargeback/elasticsearch/ingest_pipeline/usage.yml

@@ -22,6 +22,22 @@ processors:
             ctx.composite_datastream_key = ctx.composite_key + '_' + ctx.datastream;
           }
         }
+
+        // Parse Fleet data stream naming: <type>-<dataset>-<namespace>
+        // first segment → ds_type; last segment → ds_namespace; else other
+        if (ctx.datastream != null && ctx.datastream != '') {
+          def parts = ctx.datastream.splitOnToken('-');
+          if (parts.length >= 2) {
+            ctx.ds_type = parts[0];
+            ctx.ds_namespace = parts[parts.length - 1];
+          } else {
+            ctx.ds_type = 'other';
+            ctx.ds_namespace = 'other';
+          }
+        } else {
+          ctx.ds_type = 'other';
+          ctx.ds_namespace = 'other';
+        }
   - pipeline:
       description: '[Fleet] Global pipeline for all data streams'
       ignore_missing_pipeline: true

packages/chargeback/elasticsearch/transform/billing_cluster_cost/transform.yml

@@ -30,7 +30,7 @@ source:
           emit('');
 dest:
   index: billing_cluster_cost_lookup
-  pipeline: 0.4.0-billing
+  pipeline: 0.5.1-billing
 frequency: 60m
 sync:
   time:
@@ -67,4 +67,4 @@ _meta:
   run_as_kibana_system: false
   # Bump this version to delete, reinstall, and restart the transform during package.
   # Version bump is needed if there is any code change in transform.
-  fleet_transform_version: 0.4.0
+  fleet_transform_version: 0.5.1

packages/chargeback/elasticsearch/transform/billing_realized_pool/transform.yml

@@ -55,7 +55,7 @@ source:
           emit('');
 dest:
   index: billing_realized_pool_lookup
-  pipeline: 0.4.0-realized_pool
+  pipeline: 0.5.1-realized_pool
 frequency: 60m
 sync:
   time:
@@ -92,4 +92,4 @@ settings:
 _meta:
   managed: true
   run_as_kibana_system: false
-  fleet_transform_version: 0.4.0
+  fleet_transform_version: 0.5.1

packages/chargeback/elasticsearch/transform/chargeback_conf_lookup/transform.yml

@@ -57,7 +57,7 @@ source:
         source: "emit(50)"
 dest:
   index: chargeback_conf_lookup
-  pipeline: 0.4.0-chargeback_conf_lookup
+  pipeline: 0.5.1-chargeback_conf_lookup
 pivot:
   group_by:
     config_join_key:
@@ -106,4 +106,4 @@ settings:
 _meta:
   managed: true
   run_as_kibana_system: false
-  fleet_transform_version: 0.4.0
+  fleet_transform_version: 0.5.1

packages/chargeback/elasticsearch/transform/cluster_capacity_utilization/transform.yml

@@ -33,7 +33,7 @@ source:
           }
 dest:
   index: cluster_capacity_utilization_lookup
-  pipeline: 0.4.0-capacity_utilization
+  pipeline: 0.5.1-capacity_utilization
 frequency: 60m
 sync:
   time:
@@ -65,4 +65,4 @@ settings:
 _meta:
   managed: true
   run_as_kibana_system: false
-  fleet_transform_version: 0.4.0
+  fleet_transform_version: 0.5.1

packages/chargeback/elasticsearch/transform/cluster_datastream_contribution/fields/fields.yml

@@ -7,6 +7,12 @@
 - name: datastream
   type: keyword
   description: The name of the data stream.
+- name: ds_type
+  type: keyword
+  description: First segment of the data stream name (for example logs, metrics, traces). Set to other when the name does not contain a hyphen.
+- name: ds_namespace
+  type: keyword
+  description: Last segment of the data stream name (team, environment, or cost-center label). Set to other when the name does not contain a hyphen.
 - name: datastream_sum_indexing_time
   type: long
   description: Total indexing time in milliseconds for the data stream.

packages/chargeback/elasticsearch/transform/cluster_datastream_contribution/transform.yml

@@ -5,7 +5,7 @@ source:
     - "*:monitoring-indices*"
 dest:
   index: cluster_datastream_contribution_lookup
-  pipeline: 0.4.0-usage
+  pipeline: 0.5.1-usage
 frequency: 60m
 sync:
   time:
@@ -61,4 +61,4 @@ _meta:
   run_as_kibana_system: false
   # Bump this version to delete, reinstall, and restart the transform during package.
   # Version bump is needed if there is any code change in transform.
-  fleet_transform_version: 0.4.0
+  fleet_transform_version: 0.5.1

packages/chargeback/elasticsearch/transform/cluster_deployment_contribution/transform.yml

@@ -6,7 +6,7 @@ source:
     - "*:monitoring-indices*" # Same pattern on all remote clusters (CCS).
 dest:
   index: cluster_deployment_contribution_lookup
-  pipeline: 0.4.0-usage
+  pipeline: 0.5.1-usage
 frequency: 60m
 sync:
   time:
@@ -43,4 +43,4 @@ _meta:
   run_as_kibana_system: false
   # Bump this version to delete, reinstall, and restart the transform during package.
   # Version bump is needed if there is any code change in transform.
-  fleet_transform_version: 0.4.0
+  fleet_transform_version: 0.5.1

packages/chargeback/elasticsearch/transform/cluster_tier_and_ds_contribution/fields/fields.yml

@@ -10,6 +10,12 @@
 - name: datastream
   type: keyword
   description: The name of the data stream.
+- name: ds_type
+  type: keyword
+  description: First segment of the data stream name (for example logs, metrics, traces). Set to other when the name does not contain a hyphen.
+- name: ds_namespace
+  type: keyword
+  description: Last segment of the data stream name (team, environment, or cost-center label). Set to other when the name does not contain a hyphen.
 - name: tier_and_datastream_sum_indexing_time
   type: long
   description: Total indexing time in milliseconds for the tier and data stream.

packages/chargeback/elasticsearch/transform/cluster_tier_and_ds_contribution/transform.yml

@@ -5,7 +5,7 @@ source:
     - "*:monitoring-indices*"
 dest:
   index: cluster_tier_and_datastream_contribution_lookup
-  pipeline: 0.4.0-usage
+  pipeline: 0.5.1-usage
 frequency: 60m
 sync:
   time:
@@ -77,4 +77,4 @@ _meta:
   run_as_kibana_system: false
   # Bump this version to delete, reinstall, and restart the transform during package.
   # Version bump is needed if there is any code change in transform.
-  fleet_transform_version: 0.4.0
+  fleet_transform_version: 0.5.1