[Chargeback] 0.4.1 follow-up: fix empty Usage dashboard (sync @timestamp)

packages/chargeback/_dev/build/docs/README.md

@@ -34,6 +34,13 @@ Chargeback costs are presented based on a configured rate and unit, used to conv
 **Required transforms:**
 - The transform `logs-elasticsearch.index_pivot-default-{VERSION}` (from the Elasticsearch integration) must be running to aggregate usage metrics per index into `monitoring-indices`.
 
+**Install order:**
+1. Install and start **ESS Billing** (or **On-Premises Billing**) so that at least one concrete backing index matches `metrics-ess_billing.billing-*` (or the on-prem billing metrics pattern).
+2. Install and start the **Elasticsearch** integration (index pivot + node stats) on monitored deployments.
+3. Install **Chargeback**. The `chargeback_conf_lookup` bootstrap transform uses the billing metrics pattern as its source trigger. If Chargeback starts before any billing backing index exists, that transform can fail to start until billing data appears; start or `_schedule_now` the transform after billing indices exist.
+
+Billing cost transforms (`billing_cluster_cost`, `billing_realized_pool`) sync on `@timestamp`. If the Usage and Cost Allocation dashboard is empty, confirm `billing_realized_pool_lookup` has documents — that index is the primary source for usage panels.
+
 **Data flow:**
 1. ESS Billing data is collected into `metrics-ess_billing.billing-*`.
 2. Elasticsearch index usage data is aggregated into `monitoring-indices*` by the Elasticsearch integration index pivot transform.
@@ -148,9 +155,11 @@ The integration creates eight transforms to aggregate cost and usage data:
 
 **Usage transforms** (from monitoring indices):
 5. **`cluster_deployment_contribution`**: indexing, querying, and storage metrics per deployment/day.
-6. **`cluster_datastream_contribution`**: same metrics split by data stream.
+6. **`cluster_datastream_contribution`**: same metrics split by data stream. The usage ingest pipeline also sets `ds_type` (first `-` segment) and `ds_namespace` (last `-` segment) from the data stream name, or `other` when the name is non-conforming.
 7. **`cluster_tier_contribution`**: same metrics split by data tier.
-8. **`cluster_tier_and_ds_contribution`**: same metrics split by both tier and data stream.
+8. **`cluster_tier_and_ds_contribution`**: same metrics split by both tier and data stream (includes `ds_type` and `ds_namespace`).
+
+All transform destination documents include ECS `event.ingested` (when the lookup row was written).
 
 These transforms produce lookup indices queried by the dashboards using ES|QL LOOKUP JOINs.
 
@@ -191,9 +200,13 @@ Answers: *which data streams and tiers drive cost, and how efficiently are we us
 Sections:
 - **Deployment cost allocation (usage-based)**: normalized cost per deployment split by data tier (usage-weighted). Shows which deployments consume the most of their chargeable pool across tiers.
 - **Datatiers / utilization**: provisioned capacity vs chargeable pool, utilization p95 per deployment.
-- **Data tier and data stream overview**: top-20 data streams by indexing / query / storage cost, blended cost totals, workload breakdown by tier.
+- **Data tier and data stream overview**: top-20 data streams by indexing / query / storage cost, blended cost totals, blended cost by data stream **namespace** (`ds_namespace`) and **type** (`ds_type`), and workload breakdown by tier.
 - **Data tier and data stream per day**: time-series panels (indexing, querying, storage, blended) broken out by data stream and data tier (usage-based), including absolute cost and percentage share.
 
+#### Namespace-based ownership on shared deployments
+
+When several teams share one deployment, assign each team a unique Fleet **namespace** so their data lands in distinct data streams (`<type>-<dataset>-<namespace>`). Chargeback parses `ds_namespace` and `ds_type` from the full data stream name for breakdown panels. Interactive control-bar filters on those fields are not part of this release line; use the namespace and type panels (or a KQL/ES|QL query) until a later package that ships ES|QL variable controls.
+
 ![Usage and Cost Allocation](../img/chargeback-usage-allocation.png)
 
 ### [Chargeback] Configuration
@@ -215,6 +228,10 @@ Do not expect the chargeable pool to equal the full deployment bill. Non-allocat
 
 When `node_stats` is missing for a deployment/day, utilization defaults to 100%.
 
+#### Small differences versus the ESS Billing dashboard
+
+Chargeback billing and usage transforms aggregate on a **calendar day** (`calendar_interval: 1d`) with a sync delay (typically about one hour) and a periodic frequency. The ESS Billing dashboard can show near-real-time or partial-day totals for the current UTC day. Small differences for "today" or windows that cross midnight are expected. When you compare Chargeback to ESS Billing, use **completed UTC days** (or extend the billing time range across the day boundary) rather than an incomplete current day.
+
 ## Deployment Groups
 
 The integration supports organising deployments into logical groups using the `chargeback_group` tag on ESS Billing deployments. This enables cost allocation and filtering by team, project, or any organisational structure.
@@ -243,6 +260,12 @@ For more information, refer to the [Elastic documentation](https://www.elastic.c
 
 ## Upgrade Notes
 
+### Upgrading to 0.4.1
+
+1. Upgrade the Fleet package to **0.4.1**. Kibana requirement remains `^9.2.0`.
+2. Usage transforms rewrite lookup documents through the updated `usage` pipeline so new rows include `ds_type`, `ds_namespace`, and `event.ingested`. Reset usage transforms (or wait for incremental runs) if you need historical rows enriched.
+3. Confirm ESS Billing (or On-Premises Billing) has produced backing indices before relying on `chargeback_conf_lookup` after a fresh install.
+
 ### Upgrading to 0.4.0
 
 1. Upgrade the Fleet package to **0.4.0**.

packages/chargeback/changelog.yml

@@ -1,4 +1,21 @@
 # newer versions go on top
+- version: 0.4.1
+  changes:
+    - description: "Parse data stream names into ds_type (first segment) and ds_namespace (last segment) on the usage path, with other fallback for non-conforming names. Add Usage dashboard breakdown panels for blended cost by namespace and by type (no interactive control-bar filters). Document namespace-based ownership for shared deployments."
+      type: enhancement
+      link: https://github.com/elastic/elasticsearch-chargeback/issues/23
+    - description: "Add ECS event.ingested to all chargeback transform destination ingest pipelines and field definitions so lookup rows record when they were written."
+      type: enhancement
+      link: https://github.com/elastic/elasticsearch-chargeback/issues/97
+    - description: "Document chargeback_conf_lookup bootstrap install order (billing backing index must exist before Chargeback starts)."
+      type: enhancement
+      link: https://github.com/elastic/elasticsearch-chargeback/issues/96
+    - description: "Document expected small deltas versus the ESS Billing dashboard for incomplete UTC days and transform sync delay; compare completed days when reconciling."
+      type: enhancement
+      link: https://github.com/elastic/elasticsearch-chargeback/issues/66
+    - description: "Fix empty Usage and Cost Allocation dashboard when billing sources lack a mapped event.ingested field (common with On-Prem Billing). billing_cluster_cost and billing_realized_pool now sync on @timestamp so billing_realized_pool_lookup populates and usage panels can render."
+      type: bugfix
+      link: https://github.com/elastic/elasticsearch-chargeback/issues/23
 - version: 0.4.0
   changes:
     - description: "Split monolithic dashboard into three focused dashboards: '[Chargeback] Billing Components Overview' (full invoice by deployment group and billing component), '[Chargeback] Usage & Cost Allocation' (usage-based chargeable pool allocation by tier and data stream), and '[Chargeback] Configuration' (rate, weights, and date-window reference). Resolves elasticsearch-chargeback#8."

packages/chargeback/docs/README.md

@@ -34,6 +34,13 @@ Chargeback costs are presented based on a configured rate and unit, used to conv
 **Required transforms:**
 - The transform `logs-elasticsearch.index_pivot-default-{VERSION}` (from the Elasticsearch integration) must be running to aggregate usage metrics per index into `monitoring-indices`.
 
+**Install order:**
+1. Install and start **ESS Billing** (or **On-Premises Billing**) so that at least one concrete backing index matches `metrics-ess_billing.billing-*` (or the on-prem billing metrics pattern).
+2. Install and start the **Elasticsearch** integration (index pivot + node stats) on monitored deployments.
+3. Install **Chargeback**. The `chargeback_conf_lookup` bootstrap transform uses the billing metrics pattern as its source trigger. If Chargeback starts before any billing backing index exists, that transform can fail to start until billing data appears; start or `_schedule_now` the transform after billing indices exist.
+
+Billing cost transforms (`billing_cluster_cost`, `billing_realized_pool`) sync on `@timestamp`. If the Usage and Cost Allocation dashboard is empty, confirm `billing_realized_pool_lookup` has documents — that index is the primary source for usage panels.
+
 **Data flow:**
 1. ESS Billing data is collected into `metrics-ess_billing.billing-*`.
 2. Elasticsearch index usage data is aggregated into `monitoring-indices*` by the Elasticsearch integration index pivot transform.
@@ -148,9 +155,11 @@ The integration creates eight transforms to aggregate cost and usage data:
 
 **Usage transforms** (from monitoring indices):
 5. **`cluster_deployment_contribution`**: indexing, querying, and storage metrics per deployment/day.
-6. **`cluster_datastream_contribution`**: same metrics split by data stream.
+6. **`cluster_datastream_contribution`**: same metrics split by data stream. The usage ingest pipeline also sets `ds_type` (first `-` segment) and `ds_namespace` (last `-` segment) from the data stream name, or `other` when the name is non-conforming.
 7. **`cluster_tier_contribution`**: same metrics split by data tier.
-8. **`cluster_tier_and_ds_contribution`**: same metrics split by both tier and data stream.
+8. **`cluster_tier_and_ds_contribution`**: same metrics split by both tier and data stream (includes `ds_type` and `ds_namespace`).
+
+All transform destination documents include ECS `event.ingested` (when the lookup row was written).
 
 These transforms produce lookup indices queried by the dashboards using ES|QL LOOKUP JOINs.
 
@@ -191,9 +200,13 @@ Answers: *which data streams and tiers drive cost, and how efficiently are we us
 Sections:
 - **Deployment cost allocation (usage-based)**: normalized cost per deployment split by data tier (usage-weighted). Shows which deployments consume the most of their chargeable pool across tiers.
 - **Datatiers / utilization**: provisioned capacity vs chargeable pool, utilization p95 per deployment.
-- **Data tier and data stream overview**: top-20 data streams by indexing / query / storage cost, blended cost totals, workload breakdown by tier.
+- **Data tier and data stream overview**: top-20 data streams by indexing / query / storage cost, blended cost totals, blended cost by data stream **namespace** (`ds_namespace`) and **type** (`ds_type`), and workload breakdown by tier.
 - **Data tier and data stream per day**: time-series panels (indexing, querying, storage, blended) broken out by data stream and data tier (usage-based), including absolute cost and percentage share.
 
+#### Namespace-based ownership on shared deployments
+
+When several teams share one deployment, assign each team a unique Fleet **namespace** so their data lands in distinct data streams (`<type>-<dataset>-<namespace>`). Chargeback parses `ds_namespace` and `ds_type` from the full data stream name for breakdown panels. Interactive control-bar filters on those fields are not part of this release line; use the namespace and type panels (or a KQL/ES|QL query) until a later package that ships ES|QL variable controls.
+
 ![Usage and Cost Allocation](../img/chargeback-usage-allocation.png)
 
 ### [Chargeback] Configuration
@@ -215,6 +228,10 @@ Do not expect the chargeable pool to equal the full deployment bill. Non-allocat
 
 When `node_stats` is missing for a deployment/day, utilization defaults to 100%.
 
+#### Small differences versus the ESS Billing dashboard
+
+Chargeback billing and usage transforms aggregate on a **calendar day** (`calendar_interval: 1d`) with a sync delay (typically about one hour) and a periodic frequency. The ESS Billing dashboard can show near-real-time or partial-day totals for the current UTC day. Small differences for "today" or windows that cross midnight are expected. When you compare Chargeback to ESS Billing, use **completed UTC days** (or extend the billing time range across the day boundary) rather than an incomplete current day.
+
 ## Deployment Groups
 
 The integration supports organising deployments into logical groups using the `chargeback_group` tag on ESS Billing deployments. This enables cost allocation and filtering by team, project, or any organisational structure.
@@ -243,6 +260,12 @@ For more information, refer to the [Elastic documentation](https://www.elastic.c
 
 ## Upgrade Notes
 
+### Upgrading to 0.4.1
+
+1. Upgrade the Fleet package to **0.4.1**. Kibana requirement remains `^9.2.0`.
+2. Usage transforms rewrite lookup documents through the updated `usage` pipeline so new rows include `ds_type`, `ds_namespace`, and `event.ingested`. Reset usage transforms (or wait for incremental runs) if you need historical rows enriched.
+3. Confirm ESS Billing (or On-Premises Billing) has produced backing indices before relying on `chargeback_conf_lookup` after a fresh install.
+
 ### Upgrading to 0.4.0
 
 1. Upgrade the Fleet package to **0.4.0**.

packages/chargeback/elasticsearch/ingest_pipeline/billing.yml

@@ -81,6 +81,10 @@ processors:
         }
         ctx.cost_category = 'other';
         ctx.is_allocatable = false;
+  - set:
+      field: event.ingested
+      value: "{{{_ingest.timestamp}}}"
+      tag: set_event_ingested
   - pipeline:
       description: '[Fleet] Global pipeline for all data streams'
       ignore_missing_pipeline: true

packages/chargeback/elasticsearch/ingest_pipeline/capacity_utilization.yml

@@ -48,6 +48,10 @@ processors:
               ctx.storage_utilization = v / 100.0;
             }
         }
+  - set:
+      field: event.ingested
+      value: "{{{_ingest.timestamp}}}"
+      tag: set_event_ingested
   - pipeline:
       description: '[Fleet] Global pipeline for all data streams'
       ignore_missing_pipeline: true

packages/chargeback/elasticsearch/ingest_pipeline/chargeback_conf_lookup.yml

@@ -11,6 +11,10 @@ processors:
             ctx.conf_chargeable_unit_rate_unit = keys[0];
           }
         }
+  - set:
+      field: event.ingested
+      value: "{{{_ingest.timestamp}}}"
+      tag: set_event_ingested
 on_failure:
   - set:
       field: error.message

packages/chargeback/elasticsearch/ingest_pipeline/realized_pool.yml

@@ -13,6 +13,10 @@ processors:
             ctx.realized_chargeable_units = ctx.data_tier_capacity_ecu;
             ctx.total_chargeable_units = ctx.data_tier_capacity_ecu;
         }
+  - set:
+      field: event.ingested
+      value: "{{{_ingest.timestamp}}}"
+      tag: set_event_ingested
   - pipeline:
       description: '[Fleet] Global pipeline for all data streams'
       ignore_missing_pipeline: true

packages/chargeback/elasticsearch/ingest_pipeline/usage.yml

@@ -22,6 +22,26 @@ processors:
             ctx.composite_datastream_key = ctx.composite_key + '_' + ctx.datastream;
           }
         }
+
+        // Parse Fleet data stream naming: <type>-<dataset>-<namespace>
+        // first segment → ds_type; last segment → ds_namespace; else other
+        if (ctx.datastream != null && ctx.datastream != '') {
+          def parts = ctx.datastream.splitOnToken('-');
+          if (parts.length >= 2) {
+            ctx.ds_type = parts[0];
+            ctx.ds_namespace = parts[parts.length - 1];
+          } else {
+            ctx.ds_type = 'other';
+            ctx.ds_namespace = 'other';
+          }
+        } else {
+          ctx.ds_type = 'other';
+          ctx.ds_namespace = 'other';
+        }
+  - set:
+      field: event.ingested
+      value: "{{{_ingest.timestamp}}}"
+      tag: set_event_ingested
   - pipeline:
       description: '[Fleet] Global pipeline for all data streams'
       ignore_missing_pipeline: true

packages/chargeback/elasticsearch/transform/billing_cluster_cost/fields/fields.yml

@@ -31,4 +31,7 @@
   description: FinOps category (data_tier, platform, transfer, snapshot, onprem, other).
 - name: is_allocatable
   type: boolean
-  description: Whether this SKU row contributes to the realized data-tier capacity pool.
+  description: Whether this SKU row contributes to the realized data-tier capacity pool.
+- name: event.ingested
+  type: date
+  description: Timestamp when this lookup document was written to Elasticsearch.

packages/chargeback/elasticsearch/transform/billing_cluster_cost/transform.yml

@@ -30,11 +30,13 @@ source:
           emit('');
 dest:
   index: billing_cluster_cost_lookup
-  pipeline: 0.4.0-billing
+  pipeline: 0.4.1-billing
 frequency: 60m
 sync:
   time:
-    field: event.ingested
+    # Align with billing_realized_pool: @timestamp is always mapped on billing sources.
+    # event.ingested is often unset/unmapped on On-Prem Billing indices.
+    field: "@timestamp"
     delay: 1h
 pivot:
   group_by:
@@ -67,4 +69,4 @@ _meta:
   run_as_kibana_system: false
   # Bump this version to delete, reinstall, and restart the transform during package.
   # Version bump is needed if there is any code change in transform.
-  fleet_transform_version: 0.4.0
+  fleet_transform_version: 0.4.1

packages/chargeback/elasticsearch/transform/billing_realized_pool/fields/fields.yml

@@ -32,3 +32,6 @@
   type: alias
   path: total_chargeable_units
   description: Deprecated alias of total_chargeable_units.
+- name: event.ingested
+  type: date
+  description: Timestamp when this lookup document was written to Elasticsearch.

packages/chargeback/elasticsearch/transform/billing_realized_pool/transform.yml

@@ -55,11 +55,14 @@ source:
           emit('');
 dest:
   index: billing_realized_pool_lookup
-  pipeline: 0.4.0-realized_pool
+  pipeline: 0.4.1-realized_pool
 frequency: 60m
 sync:
   time:
-    field: event.ingested
+    # Use @timestamp (billing day), not event.ingested. On-Prem Billing and some
+    # seeded sources do not map event.ingested, which left this transform at 0 docs
+    # and emptied the Usage & Cost Allocation dashboard.
+    field: "@timestamp"
     delay: 1h
 pivot:
   group_by:
@@ -92,4 +95,4 @@ settings:
 _meta:
   managed: true
   run_as_kibana_system: false
-  fleet_transform_version: 0.4.0
+  fleet_transform_version: 0.4.1

packages/chargeback/elasticsearch/transform/chargeback_conf_lookup/fields/base-fields.yml

@@ -45,3 +45,6 @@
 - name: conf_storage_cost_weight
   type: integer
   description: Weight for illustrative storage split of chargeable pool in datatiers panels (default 50).
+- name: event.ingested
+  type: date
+  description: Timestamp when this lookup document was written to Elasticsearch.

packages/chargeback/elasticsearch/transform/chargeback_conf_lookup/transform.yml

@@ -57,7 +57,7 @@ source:
         source: "emit(50)"
 dest:
   index: chargeback_conf_lookup
-  pipeline: 0.4.0-chargeback_conf_lookup
+  pipeline: 0.4.1-chargeback_conf_lookup
 pivot:
   group_by:
     config_join_key:
@@ -106,4 +106,4 @@ settings:
 _meta:
   managed: true
   run_as_kibana_system: false
-  fleet_transform_version: 0.4.0
+  fleet_transform_version: 0.4.1

packages/chargeback/elasticsearch/transform/cluster_capacity_utilization/fields/fields.yml

@@ -22,3 +22,6 @@
 - name: storage_utilization
   type: double
   description: Optional normalized disk utilization (disk_used_pct_p95 / 100).
+- name: event.ingested
+  type: date
+  description: Timestamp when this lookup document was written to Elasticsearch.

packages/chargeback/elasticsearch/transform/cluster_capacity_utilization/transform.yml

@@ -33,7 +33,7 @@ source:
           }
 dest:
   index: cluster_capacity_utilization_lookup
-  pipeline: 0.4.0-capacity_utilization
+  pipeline: 0.4.1-capacity_utilization
 frequency: 60m
 sync:
   time:
@@ -65,4 +65,4 @@ settings:
 _meta:
   managed: true
   run_as_kibana_system: false
-  fleet_transform_version: 0.4.0
+  fleet_transform_version: 0.4.1

packages/chargeback/elasticsearch/transform/cluster_datastream_contribution/fields/fields.yml

@@ -7,6 +7,15 @@
 - name: datastream
   type: keyword
   description: The name of the data stream.
+- name: ds_type
+  type: keyword
+  description: First segment of the data stream name (for example logs, metrics, traces). Set to other when the name does not contain a hyphen.
+- name: ds_namespace
+  type: keyword
+  description: Last segment of the data stream name (team, environment, or cost-center label). Set to other when the name does not contain a hyphen.
+- name: event.ingested
+  type: date
+  description: Timestamp when this lookup document was written to Elasticsearch.
 - name: datastream_sum_indexing_time
   type: long
   description: Total indexing time in milliseconds for the data stream.