From f9faf025fe92ed931fbb2ed29979ddd4186e9daf Mon Sep 17 00:00:00 2001 From: Johannes Mahne Date: Mon, 3 Aug 2026 17:17:16 +0200 Subject: [PATCH 1/3] [Chargeback] 0.4.1 bulk: ds_type/ds_namespace panels, event.ingested, docs Ship low-dependency patch on kibana ^9.2: parse data stream type/namespace with Usage breakdown panels (no control-bar filters), ECS event.ingested on lookup destinations, and install/reconciliation docs for #96/#66. Co-authored-by: Cursor --- packages/chargeback/_dev/build/docs/README.md | 27 +- packages/chargeback/changelog.yml | 14 + packages/chargeback/docs/README.md | 27 +- .../elasticsearch/ingest_pipeline/billing.yml | 4 + .../ingest_pipeline/capacity_utilization.yml | 4 + .../chargeback_conf_lookup.yml | 4 + .../ingest_pipeline/realized_pool.yml | 4 + .../elasticsearch/ingest_pipeline/usage.yml | 20 + .../billing_cluster_cost/fields/fields.yml | 5 +- .../billing_cluster_cost/transform.yml | 4 +- .../billing_realized_pool/fields/fields.yml | 3 + .../billing_realized_pool/transform.yml | 4 +- .../fields/base-fields.yml | 3 + .../chargeback_conf_lookup/transform.yml | 4 +- .../fields/fields.yml | 3 + .../transform.yml | 4 +- .../fields/fields.yml | 9 + .../transform.yml | 4 +- .../fields/fields.yml | 3 + .../transform.yml | 4 +- .../fields/fields.yml | 9 + .../transform.yml | 4 +- .../fields/fields.yml | 3 + .../cluster_tier_contribution/transform.yml | 4 +- ...-21588d0e-fb6a-4f76-ad2f-cd7b3d7a3d7c.json | 7030 +++++++++-------- packages/chargeback/manifest.yml | 2 +- 26 files changed, 3771 insertions(+), 3435 deletions(-) diff --git a/packages/chargeback/_dev/build/docs/README.md b/packages/chargeback/_dev/build/docs/README.md index 43d18e9e420..ae210517652 100755 --- a/packages/chargeback/_dev/build/docs/README.md +++ b/packages/chargeback/_dev/build/docs/README.md @@ -34,6 +34,11 @@ Chargeback costs are presented based on a configured rate and unit, used to conv **Required transforms:** - The transform `logs-elasticsearch.index_pivot-default-{VERSION}` (from the Elasticsearch integration) must be running to aggregate usage metrics per index into `monitoring-indices`. +**Install order:** +1. Install and start **ESS Billing** (or **On-Premises Billing**) so that at least one concrete backing index matches `metrics-ess_billing.billing-*` (or the on-prem billing metrics pattern). +2. Install and start the **Elasticsearch** integration (index pivot + node stats) on monitored deployments. +3. Install **Chargeback**. The `chargeback_conf_lookup` bootstrap transform uses the billing metrics pattern as its source trigger. If Chargeback starts before any billing backing index exists, that transform can fail to start until billing data appears; start or `_schedule_now` the transform after billing indices exist. + **Data flow:** 1. ESS Billing data is collected into `metrics-ess_billing.billing-*`. 2. Elasticsearch index usage data is aggregated into `monitoring-indices*` by the Elasticsearch integration index pivot transform. @@ -148,9 +153,11 @@ The integration creates eight transforms to aggregate cost and usage data: **Usage transforms** (from monitoring indices): 5. **`cluster_deployment_contribution`**: indexing, querying, and storage metrics per deployment/day. -6. **`cluster_datastream_contribution`**: same metrics split by data stream. +6. **`cluster_datastream_contribution`**: same metrics split by data stream. The usage ingest pipeline also sets `ds_type` (first `-` segment) and `ds_namespace` (last `-` segment) from the data stream name, or `other` when the name is non-conforming. 7. **`cluster_tier_contribution`**: same metrics split by data tier. -8. **`cluster_tier_and_ds_contribution`**: same metrics split by both tier and data stream. +8. **`cluster_tier_and_ds_contribution`**: same metrics split by both tier and data stream (includes `ds_type` and `ds_namespace`). + +All transform destination documents include ECS `event.ingested` (when the lookup row was written). These transforms produce lookup indices queried by the dashboards using ES|QL LOOKUP JOINs. @@ -191,9 +198,13 @@ Answers: *which data streams and tiers drive cost, and how efficiently are we us Sections: - **Deployment cost allocation (usage-based)**: normalized cost per deployment split by data tier (usage-weighted). Shows which deployments consume the most of their chargeable pool across tiers. - **Datatiers / utilization**: provisioned capacity vs chargeable pool, utilization p95 per deployment. -- **Data tier and data stream overview**: top-20 data streams by indexing / query / storage cost, blended cost totals, workload breakdown by tier. +- **Data tier and data stream overview**: top-20 data streams by indexing / query / storage cost, blended cost totals, blended cost by data stream **namespace** (`ds_namespace`) and **type** (`ds_type`), and workload breakdown by tier. - **Data tier and data stream per day**: time-series panels (indexing, querying, storage, blended) broken out by data stream and data tier (usage-based), including absolute cost and percentage share. +#### Namespace-based ownership on shared deployments + +When several teams share one deployment, assign each team a unique Fleet **namespace** so their data lands in distinct data streams (`--`). Chargeback parses `ds_namespace` and `ds_type` from the full data stream name for breakdown panels. Interactive control-bar filters on those fields are not part of this release line; use the namespace and type panels (or a KQL/ES|QL query) until a later package that ships ES|QL variable controls. + ![Usage and Cost Allocation](../img/chargeback-usage-allocation.png) ### [Chargeback] Configuration @@ -215,6 +226,10 @@ Do not expect the chargeable pool to equal the full deployment bill. Non-allocat When `node_stats` is missing for a deployment/day, utilization defaults to 100%. +#### Small differences versus the ESS Billing dashboard + +Chargeback billing and usage transforms aggregate on a **calendar day** (`calendar_interval: 1d`) with a sync delay (typically about one hour) and a periodic frequency. The ESS Billing dashboard can show near-real-time or partial-day totals for the current UTC day. Small differences for "today" or windows that cross midnight are expected. When you compare Chargeback to ESS Billing, use **completed UTC days** (or extend the billing time range across the day boundary) rather than an incomplete current day. + ## Deployment Groups The integration supports organising deployments into logical groups using the `chargeback_group` tag on ESS Billing deployments. This enables cost allocation and filtering by team, project, or any organisational structure. @@ -243,6 +258,12 @@ For more information, refer to the [Elastic documentation](https://www.elastic.c ## Upgrade Notes +### Upgrading to 0.4.1 + +1. Upgrade the Fleet package to **0.4.1**. Kibana requirement remains `^9.2.0`. +2. Usage transforms rewrite lookup documents through the updated `usage` pipeline so new rows include `ds_type`, `ds_namespace`, and `event.ingested`. Reset usage transforms (or wait for incremental runs) if you need historical rows enriched. +3. Confirm ESS Billing (or On-Premises Billing) has produced backing indices before relying on `chargeback_conf_lookup` after a fresh install. + ### Upgrading to 0.4.0 1. Upgrade the Fleet package to **0.4.0**. diff --git a/packages/chargeback/changelog.yml b/packages/chargeback/changelog.yml index 3b7f64721be..ad6e949a452 100644 --- a/packages/chargeback/changelog.yml +++ b/packages/chargeback/changelog.yml @@ -1,4 +1,18 @@ # newer versions go on top +- version: 0.4.1 + changes: + - description: "Parse data stream names into ds_type (first segment) and ds_namespace (last segment) on the usage path, with other fallback for non-conforming names. Add Usage dashboard breakdown panels for blended cost by namespace and by type (no interactive control-bar filters). Document namespace-based ownership for shared deployments." + type: enhancement + link: https://github.com/elastic/elasticsearch-chargeback/issues/23 + - description: "Add ECS event.ingested to all chargeback transform destination ingest pipelines and field definitions so lookup rows record when they were written." + type: enhancement + link: https://github.com/elastic/elasticsearch-chargeback/issues/97 + - description: "Document chargeback_conf_lookup bootstrap install order (billing backing index must exist before Chargeback starts)." + type: enhancement + link: https://github.com/elastic/elasticsearch-chargeback/issues/96 + - description: "Document expected small deltas versus the ESS Billing dashboard for incomplete UTC days and transform sync delay; compare completed days when reconciling." + type: enhancement + link: https://github.com/elastic/elasticsearch-chargeback/issues/66 - version: 0.4.0 changes: - description: "Split monolithic dashboard into three focused dashboards: '[Chargeback] Billing Components Overview' (full invoice by deployment group and billing component), '[Chargeback] Usage & Cost Allocation' (usage-based chargeable pool allocation by tier and data stream), and '[Chargeback] Configuration' (rate, weights, and date-window reference). Resolves elasticsearch-chargeback#8." diff --git a/packages/chargeback/docs/README.md b/packages/chargeback/docs/README.md index 43d18e9e420..ae210517652 100644 --- a/packages/chargeback/docs/README.md +++ b/packages/chargeback/docs/README.md @@ -34,6 +34,11 @@ Chargeback costs are presented based on a configured rate and unit, used to conv **Required transforms:** - The transform `logs-elasticsearch.index_pivot-default-{VERSION}` (from the Elasticsearch integration) must be running to aggregate usage metrics per index into `monitoring-indices`. +**Install order:** +1. Install and start **ESS Billing** (or **On-Premises Billing**) so that at least one concrete backing index matches `metrics-ess_billing.billing-*` (or the on-prem billing metrics pattern). +2. Install and start the **Elasticsearch** integration (index pivot + node stats) on monitored deployments. +3. Install **Chargeback**. The `chargeback_conf_lookup` bootstrap transform uses the billing metrics pattern as its source trigger. If Chargeback starts before any billing backing index exists, that transform can fail to start until billing data appears; start or `_schedule_now` the transform after billing indices exist. + **Data flow:** 1. ESS Billing data is collected into `metrics-ess_billing.billing-*`. 2. Elasticsearch index usage data is aggregated into `monitoring-indices*` by the Elasticsearch integration index pivot transform. @@ -148,9 +153,11 @@ The integration creates eight transforms to aggregate cost and usage data: **Usage transforms** (from monitoring indices): 5. **`cluster_deployment_contribution`**: indexing, querying, and storage metrics per deployment/day. -6. **`cluster_datastream_contribution`**: same metrics split by data stream. +6. **`cluster_datastream_contribution`**: same metrics split by data stream. The usage ingest pipeline also sets `ds_type` (first `-` segment) and `ds_namespace` (last `-` segment) from the data stream name, or `other` when the name is non-conforming. 7. **`cluster_tier_contribution`**: same metrics split by data tier. -8. **`cluster_tier_and_ds_contribution`**: same metrics split by both tier and data stream. +8. **`cluster_tier_and_ds_contribution`**: same metrics split by both tier and data stream (includes `ds_type` and `ds_namespace`). + +All transform destination documents include ECS `event.ingested` (when the lookup row was written). These transforms produce lookup indices queried by the dashboards using ES|QL LOOKUP JOINs. @@ -191,9 +198,13 @@ Answers: *which data streams and tiers drive cost, and how efficiently are we us Sections: - **Deployment cost allocation (usage-based)**: normalized cost per deployment split by data tier (usage-weighted). Shows which deployments consume the most of their chargeable pool across tiers. - **Datatiers / utilization**: provisioned capacity vs chargeable pool, utilization p95 per deployment. -- **Data tier and data stream overview**: top-20 data streams by indexing / query / storage cost, blended cost totals, workload breakdown by tier. +- **Data tier and data stream overview**: top-20 data streams by indexing / query / storage cost, blended cost totals, blended cost by data stream **namespace** (`ds_namespace`) and **type** (`ds_type`), and workload breakdown by tier. - **Data tier and data stream per day**: time-series panels (indexing, querying, storage, blended) broken out by data stream and data tier (usage-based), including absolute cost and percentage share. +#### Namespace-based ownership on shared deployments + +When several teams share one deployment, assign each team a unique Fleet **namespace** so their data lands in distinct data streams (`--`). Chargeback parses `ds_namespace` and `ds_type` from the full data stream name for breakdown panels. Interactive control-bar filters on those fields are not part of this release line; use the namespace and type panels (or a KQL/ES|QL query) until a later package that ships ES|QL variable controls. + ![Usage and Cost Allocation](../img/chargeback-usage-allocation.png) ### [Chargeback] Configuration @@ -215,6 +226,10 @@ Do not expect the chargeable pool to equal the full deployment bill. Non-allocat When `node_stats` is missing for a deployment/day, utilization defaults to 100%. +#### Small differences versus the ESS Billing dashboard + +Chargeback billing and usage transforms aggregate on a **calendar day** (`calendar_interval: 1d`) with a sync delay (typically about one hour) and a periodic frequency. The ESS Billing dashboard can show near-real-time or partial-day totals for the current UTC day. Small differences for "today" or windows that cross midnight are expected. When you compare Chargeback to ESS Billing, use **completed UTC days** (or extend the billing time range across the day boundary) rather than an incomplete current day. + ## Deployment Groups The integration supports organising deployments into logical groups using the `chargeback_group` tag on ESS Billing deployments. This enables cost allocation and filtering by team, project, or any organisational structure. @@ -243,6 +258,12 @@ For more information, refer to the [Elastic documentation](https://www.elastic.c ## Upgrade Notes +### Upgrading to 0.4.1 + +1. Upgrade the Fleet package to **0.4.1**. Kibana requirement remains `^9.2.0`. +2. Usage transforms rewrite lookup documents through the updated `usage` pipeline so new rows include `ds_type`, `ds_namespace`, and `event.ingested`. Reset usage transforms (or wait for incremental runs) if you need historical rows enriched. +3. Confirm ESS Billing (or On-Premises Billing) has produced backing indices before relying on `chargeback_conf_lookup` after a fresh install. + ### Upgrading to 0.4.0 1. Upgrade the Fleet package to **0.4.0**. diff --git a/packages/chargeback/elasticsearch/ingest_pipeline/billing.yml b/packages/chargeback/elasticsearch/ingest_pipeline/billing.yml index be179efa642..bd34f923b38 100644 --- a/packages/chargeback/elasticsearch/ingest_pipeline/billing.yml +++ b/packages/chargeback/elasticsearch/ingest_pipeline/billing.yml @@ -81,6 +81,10 @@ processors: } ctx.cost_category = 'other'; ctx.is_allocatable = false; + - set: + field: event.ingested + value: "{{{_ingest.timestamp}}}" + tag: set_event_ingested - pipeline: description: '[Fleet] Global pipeline for all data streams' ignore_missing_pipeline: true diff --git a/packages/chargeback/elasticsearch/ingest_pipeline/capacity_utilization.yml b/packages/chargeback/elasticsearch/ingest_pipeline/capacity_utilization.yml index ac46fe21361..17de72592de 100644 --- a/packages/chargeback/elasticsearch/ingest_pipeline/capacity_utilization.yml +++ b/packages/chargeback/elasticsearch/ingest_pipeline/capacity_utilization.yml @@ -48,6 +48,10 @@ processors: ctx.storage_utilization = v / 100.0; } } + - set: + field: event.ingested + value: "{{{_ingest.timestamp}}}" + tag: set_event_ingested - pipeline: description: '[Fleet] Global pipeline for all data streams' ignore_missing_pipeline: true diff --git a/packages/chargeback/elasticsearch/ingest_pipeline/chargeback_conf_lookup.yml b/packages/chargeback/elasticsearch/ingest_pipeline/chargeback_conf_lookup.yml index 380cf807e64..69d7cd5e0f3 100644 --- a/packages/chargeback/elasticsearch/ingest_pipeline/chargeback_conf_lookup.yml +++ b/packages/chargeback/elasticsearch/ingest_pipeline/chargeback_conf_lookup.yml @@ -11,6 +11,10 @@ processors: ctx.conf_chargeable_unit_rate_unit = keys[0]; } } + - set: + field: event.ingested + value: "{{{_ingest.timestamp}}}" + tag: set_event_ingested on_failure: - set: field: error.message diff --git a/packages/chargeback/elasticsearch/ingest_pipeline/realized_pool.yml b/packages/chargeback/elasticsearch/ingest_pipeline/realized_pool.yml index 805c708fc98..c3571274f66 100644 --- a/packages/chargeback/elasticsearch/ingest_pipeline/realized_pool.yml +++ b/packages/chargeback/elasticsearch/ingest_pipeline/realized_pool.yml @@ -13,6 +13,10 @@ processors: ctx.realized_chargeable_units = ctx.data_tier_capacity_ecu; ctx.total_chargeable_units = ctx.data_tier_capacity_ecu; } + - set: + field: event.ingested + value: "{{{_ingest.timestamp}}}" + tag: set_event_ingested - pipeline: description: '[Fleet] Global pipeline for all data streams' ignore_missing_pipeline: true diff --git a/packages/chargeback/elasticsearch/ingest_pipeline/usage.yml b/packages/chargeback/elasticsearch/ingest_pipeline/usage.yml index 81ad97a5616..29c014bfde1 100644 --- a/packages/chargeback/elasticsearch/ingest_pipeline/usage.yml +++ b/packages/chargeback/elasticsearch/ingest_pipeline/usage.yml @@ -22,6 +22,26 @@ processors: ctx.composite_datastream_key = ctx.composite_key + '_' + ctx.datastream; } } + + // Parse Fleet data stream naming: -- + // first segment → ds_type; last segment → ds_namespace; else other + if (ctx.datastream != null && ctx.datastream != '') { + def parts = ctx.datastream.splitOnToken('-'); + if (parts.length >= 2) { + ctx.ds_type = parts[0]; + ctx.ds_namespace = parts[parts.length - 1]; + } else { + ctx.ds_type = 'other'; + ctx.ds_namespace = 'other'; + } + } else { + ctx.ds_type = 'other'; + ctx.ds_namespace = 'other'; + } + - set: + field: event.ingested + value: "{{{_ingest.timestamp}}}" + tag: set_event_ingested - pipeline: description: '[Fleet] Global pipeline for all data streams' ignore_missing_pipeline: true diff --git a/packages/chargeback/elasticsearch/transform/billing_cluster_cost/fields/fields.yml b/packages/chargeback/elasticsearch/transform/billing_cluster_cost/fields/fields.yml index 2c6fbe1fc3a..926f872db80 100644 --- a/packages/chargeback/elasticsearch/transform/billing_cluster_cost/fields/fields.yml +++ b/packages/chargeback/elasticsearch/transform/billing_cluster_cost/fields/fields.yml @@ -31,4 +31,7 @@ description: FinOps category (data_tier, platform, transfer, snapshot, onprem, other). - name: is_allocatable type: boolean - description: Whether this SKU row contributes to the realized data-tier capacity pool. \ No newline at end of file + description: Whether this SKU row contributes to the realized data-tier capacity pool. +- name: event.ingested + type: date + description: Timestamp when this lookup document was written to Elasticsearch. diff --git a/packages/chargeback/elasticsearch/transform/billing_cluster_cost/transform.yml b/packages/chargeback/elasticsearch/transform/billing_cluster_cost/transform.yml index f7b7f8c4eac..57a472d4922 100644 --- a/packages/chargeback/elasticsearch/transform/billing_cluster_cost/transform.yml +++ b/packages/chargeback/elasticsearch/transform/billing_cluster_cost/transform.yml @@ -30,7 +30,7 @@ source: emit(''); dest: index: billing_cluster_cost_lookup - pipeline: 0.4.0-billing + pipeline: 0.4.1-billing frequency: 60m sync: time: @@ -67,4 +67,4 @@ _meta: run_as_kibana_system: false # Bump this version to delete, reinstall, and restart the transform during package. # Version bump is needed if there is any code change in transform. - fleet_transform_version: 0.4.0 + fleet_transform_version: 0.4.1 diff --git a/packages/chargeback/elasticsearch/transform/billing_realized_pool/fields/fields.yml b/packages/chargeback/elasticsearch/transform/billing_realized_pool/fields/fields.yml index b59f04262af..a0a8453dff8 100644 --- a/packages/chargeback/elasticsearch/transform/billing_realized_pool/fields/fields.yml +++ b/packages/chargeback/elasticsearch/transform/billing_realized_pool/fields/fields.yml @@ -32,3 +32,6 @@ type: alias path: total_chargeable_units description: Deprecated alias of total_chargeable_units. +- name: event.ingested + type: date + description: Timestamp when this lookup document was written to Elasticsearch. diff --git a/packages/chargeback/elasticsearch/transform/billing_realized_pool/transform.yml b/packages/chargeback/elasticsearch/transform/billing_realized_pool/transform.yml index 2aa1134935b..bb61fc8dab9 100644 --- a/packages/chargeback/elasticsearch/transform/billing_realized_pool/transform.yml +++ b/packages/chargeback/elasticsearch/transform/billing_realized_pool/transform.yml @@ -55,7 +55,7 @@ source: emit(''); dest: index: billing_realized_pool_lookup - pipeline: 0.4.0-realized_pool + pipeline: 0.4.1-realized_pool frequency: 60m sync: time: @@ -92,4 +92,4 @@ settings: _meta: managed: true run_as_kibana_system: false - fleet_transform_version: 0.4.0 + fleet_transform_version: 0.4.1 diff --git a/packages/chargeback/elasticsearch/transform/chargeback_conf_lookup/fields/base-fields.yml b/packages/chargeback/elasticsearch/transform/chargeback_conf_lookup/fields/base-fields.yml index efc9bfafc08..082b99a5cf1 100644 --- a/packages/chargeback/elasticsearch/transform/chargeback_conf_lookup/fields/base-fields.yml +++ b/packages/chargeback/elasticsearch/transform/chargeback_conf_lookup/fields/base-fields.yml @@ -45,3 +45,6 @@ - name: conf_storage_cost_weight type: integer description: Weight for illustrative storage split of chargeable pool in datatiers panels (default 50). +- name: event.ingested + type: date + description: Timestamp when this lookup document was written to Elasticsearch. diff --git a/packages/chargeback/elasticsearch/transform/chargeback_conf_lookup/transform.yml b/packages/chargeback/elasticsearch/transform/chargeback_conf_lookup/transform.yml index c4751a5c65e..32a978337af 100644 --- a/packages/chargeback/elasticsearch/transform/chargeback_conf_lookup/transform.yml +++ b/packages/chargeback/elasticsearch/transform/chargeback_conf_lookup/transform.yml @@ -57,7 +57,7 @@ source: source: "emit(50)" dest: index: chargeback_conf_lookup - pipeline: 0.4.0-chargeback_conf_lookup + pipeline: 0.4.1-chargeback_conf_lookup pivot: group_by: config_join_key: @@ -106,4 +106,4 @@ settings: _meta: managed: true run_as_kibana_system: false - fleet_transform_version: 0.4.0 + fleet_transform_version: 0.4.1 diff --git a/packages/chargeback/elasticsearch/transform/cluster_capacity_utilization/fields/fields.yml b/packages/chargeback/elasticsearch/transform/cluster_capacity_utilization/fields/fields.yml index 8c230f66aa2..598eb650917 100644 --- a/packages/chargeback/elasticsearch/transform/cluster_capacity_utilization/fields/fields.yml +++ b/packages/chargeback/elasticsearch/transform/cluster_capacity_utilization/fields/fields.yml @@ -22,3 +22,6 @@ - name: storage_utilization type: double description: Optional normalized disk utilization (disk_used_pct_p95 / 100). +- name: event.ingested + type: date + description: Timestamp when this lookup document was written to Elasticsearch. diff --git a/packages/chargeback/elasticsearch/transform/cluster_capacity_utilization/transform.yml b/packages/chargeback/elasticsearch/transform/cluster_capacity_utilization/transform.yml index 63d25a366d3..bf436923ce0 100644 --- a/packages/chargeback/elasticsearch/transform/cluster_capacity_utilization/transform.yml +++ b/packages/chargeback/elasticsearch/transform/cluster_capacity_utilization/transform.yml @@ -33,7 +33,7 @@ source: } dest: index: cluster_capacity_utilization_lookup - pipeline: 0.4.0-capacity_utilization + pipeline: 0.4.1-capacity_utilization frequency: 60m sync: time: @@ -65,4 +65,4 @@ settings: _meta: managed: true run_as_kibana_system: false - fleet_transform_version: 0.4.0 + fleet_transform_version: 0.4.1 diff --git a/packages/chargeback/elasticsearch/transform/cluster_datastream_contribution/fields/fields.yml b/packages/chargeback/elasticsearch/transform/cluster_datastream_contribution/fields/fields.yml index f489ab2bbf6..f129baf2516 100644 --- a/packages/chargeback/elasticsearch/transform/cluster_datastream_contribution/fields/fields.yml +++ b/packages/chargeback/elasticsearch/transform/cluster_datastream_contribution/fields/fields.yml @@ -7,6 +7,15 @@ - name: datastream type: keyword description: The name of the data stream. +- name: ds_type + type: keyword + description: First segment of the data stream name (for example logs, metrics, traces). Set to other when the name does not contain a hyphen. +- name: ds_namespace + type: keyword + description: Last segment of the data stream name (team, environment, or cost-center label). Set to other when the name does not contain a hyphen. +- name: event.ingested + type: date + description: Timestamp when this lookup document was written to Elasticsearch. - name: datastream_sum_indexing_time type: long description: Total indexing time in milliseconds for the data stream. diff --git a/packages/chargeback/elasticsearch/transform/cluster_datastream_contribution/transform.yml b/packages/chargeback/elasticsearch/transform/cluster_datastream_contribution/transform.yml index aca2f29639f..81f5ec52194 100644 --- a/packages/chargeback/elasticsearch/transform/cluster_datastream_contribution/transform.yml +++ b/packages/chargeback/elasticsearch/transform/cluster_datastream_contribution/transform.yml @@ -5,7 +5,7 @@ source: - "*:monitoring-indices*" dest: index: cluster_datastream_contribution_lookup - pipeline: 0.4.0-usage + pipeline: 0.4.1-usage frequency: 60m sync: time: @@ -61,4 +61,4 @@ _meta: run_as_kibana_system: false # Bump this version to delete, reinstall, and restart the transform during package. # Version bump is needed if there is any code change in transform. - fleet_transform_version: 0.4.0 + fleet_transform_version: 0.4.1 diff --git a/packages/chargeback/elasticsearch/transform/cluster_deployment_contribution/fields/fields.yml b/packages/chargeback/elasticsearch/transform/cluster_deployment_contribution/fields/fields.yml index 63866dda2e0..094144037d5 100644 --- a/packages/chargeback/elasticsearch/transform/cluster_deployment_contribution/fields/fields.yml +++ b/packages/chargeback/elasticsearch/transform/cluster_deployment_contribution/fields/fields.yml @@ -19,3 +19,6 @@ - name: composite_key type: keyword description: Composite key used for billing attribution (set by ingest pipeline) consisting of date and deployment ID. +- name: event.ingested + type: date + description: Timestamp when this lookup document was written to Elasticsearch. diff --git a/packages/chargeback/elasticsearch/transform/cluster_deployment_contribution/transform.yml b/packages/chargeback/elasticsearch/transform/cluster_deployment_contribution/transform.yml index 6e19d3e660c..5c9f83ad76f 100644 --- a/packages/chargeback/elasticsearch/transform/cluster_deployment_contribution/transform.yml +++ b/packages/chargeback/elasticsearch/transform/cluster_deployment_contribution/transform.yml @@ -6,7 +6,7 @@ source: - "*:monitoring-indices*" # Same pattern on all remote clusters (CCS). dest: index: cluster_deployment_contribution_lookup - pipeline: 0.4.0-usage + pipeline: 0.4.1-usage frequency: 60m sync: time: @@ -43,4 +43,4 @@ _meta: run_as_kibana_system: false # Bump this version to delete, reinstall, and restart the transform during package. # Version bump is needed if there is any code change in transform. - fleet_transform_version: 0.4.0 \ No newline at end of file + fleet_transform_version: 0.4.1 \ No newline at end of file diff --git a/packages/chargeback/elasticsearch/transform/cluster_tier_and_ds_contribution/fields/fields.yml b/packages/chargeback/elasticsearch/transform/cluster_tier_and_ds_contribution/fields/fields.yml index 54cc1c6bb6c..e4e2cd68564 100644 --- a/packages/chargeback/elasticsearch/transform/cluster_tier_and_ds_contribution/fields/fields.yml +++ b/packages/chargeback/elasticsearch/transform/cluster_tier_and_ds_contribution/fields/fields.yml @@ -10,6 +10,15 @@ - name: datastream type: keyword description: The name of the data stream. +- name: ds_type + type: keyword + description: First segment of the data stream name (for example logs, metrics, traces). Set to other when the name does not contain a hyphen. +- name: ds_namespace + type: keyword + description: Last segment of the data stream name (team, environment, or cost-center label). Set to other when the name does not contain a hyphen. +- name: event.ingested + type: date + description: Timestamp when this lookup document was written to Elasticsearch. - name: tier_and_datastream_sum_indexing_time type: long description: Total indexing time in milliseconds for the tier and data stream. diff --git a/packages/chargeback/elasticsearch/transform/cluster_tier_and_ds_contribution/transform.yml b/packages/chargeback/elasticsearch/transform/cluster_tier_and_ds_contribution/transform.yml index 056d88725c8..9a1f57219d7 100644 --- a/packages/chargeback/elasticsearch/transform/cluster_tier_and_ds_contribution/transform.yml +++ b/packages/chargeback/elasticsearch/transform/cluster_tier_and_ds_contribution/transform.yml @@ -5,7 +5,7 @@ source: - "*:monitoring-indices*" dest: index: cluster_tier_and_datastream_contribution_lookup - pipeline: 0.4.0-usage + pipeline: 0.4.1-usage frequency: 60m sync: time: @@ -77,4 +77,4 @@ _meta: run_as_kibana_system: false # Bump this version to delete, reinstall, and restart the transform during package. # Version bump is needed if there is any code change in transform. - fleet_transform_version: 0.4.0 \ No newline at end of file + fleet_transform_version: 0.4.1 \ No newline at end of file diff --git a/packages/chargeback/elasticsearch/transform/cluster_tier_contribution/fields/fields.yml b/packages/chargeback/elasticsearch/transform/cluster_tier_contribution/fields/fields.yml index 548f3e8f290..6f7383c14ce 100644 --- a/packages/chargeback/elasticsearch/transform/cluster_tier_contribution/fields/fields.yml +++ b/packages/chargeback/elasticsearch/transform/cluster_tier_contribution/fields/fields.yml @@ -22,3 +22,6 @@ - name: composite_key type: keyword description: Composite key used for billing attribution (set by ingest pipeline) consisting of date and data tier. +- name: event.ingested + type: date + description: Timestamp when this lookup document was written to Elasticsearch. diff --git a/packages/chargeback/elasticsearch/transform/cluster_tier_contribution/transform.yml b/packages/chargeback/elasticsearch/transform/cluster_tier_contribution/transform.yml index aa04c2a811d..faa89aedc98 100644 --- a/packages/chargeback/elasticsearch/transform/cluster_tier_contribution/transform.yml +++ b/packages/chargeback/elasticsearch/transform/cluster_tier_contribution/transform.yml @@ -5,7 +5,7 @@ source: - "*:monitoring-indices*" dest: index: cluster_tier_contribution_lookup - pipeline: 0.4.0-usage + pipeline: 0.4.1-usage frequency: 60m sync: time: @@ -59,4 +59,4 @@ _meta: run_as_kibana_system: false # Bump this version to delete, reinstall, and restart the transform during package. # Version bump is needed if there is any code change in transform. - fleet_transform_version: 0.4.0 \ No newline at end of file + fleet_transform_version: 0.4.1 \ No newline at end of file diff --git a/packages/chargeback/kibana/dashboard/chargeback-21588d0e-fb6a-4f76-ad2f-cd7b3d7a3d7c.json b/packages/chargeback/kibana/dashboard/chargeback-21588d0e-fb6a-4f76-ad2f-cd7b3d7a3d7c.json index 76c7fd817ec..6fcc89ed421 100644 --- a/packages/chargeback/kibana/dashboard/chargeback-21588d0e-fb6a-4f76-ad2f-cd7b3d7a3d7c.json +++ b/packages/chargeback/kibana/dashboard/chargeback-21588d0e-fb6a-4f76-ad2f-cd7b3d7a3d7c.json @@ -1,3539 +1,3747 @@ { - "attributes": { - "controlGroupInput": { - "chainingSystem": "HIERARCHICAL", - "controlStyle": "oneLine", - "ignoreParentSettingsJSON": { - "ignoreFilters": false, - "ignoreQuery": false, - "ignoreTimerange": false, - "ignoreValidations": false + "attributes": { + "controlGroupInput": { + "chainingSystem": "HIERARCHICAL", + "controlStyle": "oneLine", + "ignoreParentSettingsJSON": { + "ignoreFilters": false, + "ignoreQuery": false, + "ignoreTimerange": false, + "ignoreValidations": false + }, + "panelsJSON": { + "2c7df183-8a05-4ee0-b7db-415e874636cd": { + "explicitInput": { + "dataViewId": "chargeback_integration", + "exclude": false, + "existsSelected": false, + "fieldName": "deployment_name", + "searchTechnique": "prefix", + "selectedOptions": [], + "sort": { + "by": "_count", + "direction": "desc" }, - "panelsJSON": { - "2c7df183-8a05-4ee0-b7db-415e874636cd": { - "explicitInput": { - "dataViewId": "chargeback_integration", - "exclude": false, - "existsSelected": false, - "fieldName": "deployment_name", - "searchTechnique": "prefix", - "selectedOptions": [], - "sort": { - "by": "_count", - "direction": "desc" - }, - "title": "Deployment name", - "id": "2c7df183-8a05-4ee0-b7db-415e874636cd" - }, - "grow": false, - "order": 1, - "type": "optionsListControl", - "width": "medium" - }, - "b680ef70-4697-41e5-bf5f-e7e1e700ec3e": { - "explicitInput": { - "dataViewId": "chargeback_integration", - "exclude": false, - "existsSelected": false, - "fieldName": "deployment_group", - "searchTechnique": "prefix", - "selectedOptions": [], - "sort": { - "by": "_count", - "direction": "desc" - }, - "title": "Deployment group", - "id": "b680ef70-4697-41e5-bf5f-e7e1e700ec3e" - }, - "grow": false, - "order": 0, - "type": "optionsListControl", - "width": "medium" - } + "title": "Deployment name", + "id": "2c7df183-8a05-4ee0-b7db-415e874636cd" + }, + "grow": false, + "order": 1, + "type": "optionsListControl", + "width": "medium" + }, + "b680ef70-4697-41e5-bf5f-e7e1e700ec3e": { + "explicitInput": { + "dataViewId": "chargeback_integration", + "exclude": false, + "existsSelected": false, + "fieldName": "deployment_group", + "searchTechnique": "prefix", + "selectedOptions": [], + "sort": { + "by": "_count", + "direction": "desc" + }, + "title": "Deployment group", + "id": "b680ef70-4697-41e5-bf5f-e7e1e700ec3e" + }, + "grow": false, + "order": 0, + "type": "optionsListControl", + "width": "medium" + } + }, + "showApplySelections": false + }, + "description": "Realized cost allocation per data tier and data stream. Shows utilization, top consumers, and time-series cost breakdowns.", + "kibanaSavedObjectMeta": { + "searchSourceJSON": { + "filter": [], + "query": { + "language": "kuery", + "query": "" + } + } + }, + "optionsJSON": { + "hidePanelTitles": false, + "syncColors": false, + "syncCursor": true, + "syncTooltips": false, + "useMargins": true + }, + "panelsJSON": [ + { + "embeddableConfig": { + "enhancements": {}, + "layout": "horizontal", + "links": [ + { + "destinationRefName": "link_ea0417fa-630d-c656-8f4b-a8d7b9916b29_dashboard", + "id": "ea0417fa-630d-c656-8f4b-a8d7b9916b29", + "label": "[Chargeback] Billing Components Overview", + "options": { + "openInNewTab": false, + "useCurrentDateRange": true, + "useCurrentFilters": true + }, + "order": 0, + "type": "dashboardLink" + }, + { + "destinationRefName": "link_dc802531-4211-c74b-d9db-b197b528bbc8_dashboard", + "id": "dc802531-4211-c74b-d9db-b197b528bbc8", + "label": "[Chargeback] Usage & Cost Allocation", + "options": { + "openInNewTab": false, + "useCurrentDateRange": true, + "useCurrentFilters": true + }, + "order": 1, + "type": "dashboardLink" }, - "showApplySelections": false + { + "destinationRefName": "link_4e06e36e-f5d7-981f-8554-57756d1c59b8_dashboard", + "id": "4e06e36e-f5d7-981f-8554-57756d1c59b8", + "label": "[Chargeback] Configuration", + "options": { + "openInNewTab": false, + "useCurrentDateRange": true, + "useCurrentFilters": true + }, + "order": 2, + "type": "dashboardLink" + } + ] + }, + "gridData": { + "h": 2, + "i": "13cd2a6a-ec89-4553-ae64-cbc347770aa5", + "w": 48, + "x": 0, + "y": 0 }, - "description": "Realized cost allocation per data tier and data stream. Shows utilization, top consumers, and time-series cost breakdowns.", - "kibanaSavedObjectMeta": { - "searchSourceJSON": { + "panelIndex": "13cd2a6a-ec89-4553-ae64-cbc347770aa5", + "type": "links" + }, + { + "embeddableConfig": { + "enhancements": { + "dynamicActions": { + "events": [] + } + }, + "savedVis": { + "data": { + "aggs": [], + "searchSource": { "filter": [], "query": { - "language": "kuery", - "query": "" + "language": "kuery", + "query": "" } - } + } + }, + "description": "", + "id": "", + "params": { + "fontSize": 12, + "markdown": "# Data Tiers / Utilization", + "openLinksInNewTab": false + }, + "title": "", + "type": "markdown", + "uiState": {} + } }, - "optionsJSON": { - "hidePanelTitles": false, - "syncColors": false, - "syncCursor": true, - "syncTooltips": false, - "useMargins": true + "gridData": { + "h": 3, + "i": "eca9d216-830e-4509-88c7-66b1dd354c71", + "sectionId": "0fceb22c-1e73-45e4-b634-006f6dcc5f43", + "w": 48, + "x": 0, + "y": 0 }, - "panelsJSON": [ - { - "embeddableConfig": { - "enhancements": {}, - "layout": "horizontal", - "links": [ - { - "destinationRefName": "link_ea0417fa-630d-c656-8f4b-a8d7b9916b29_dashboard", - "id": "ea0417fa-630d-c656-8f4b-a8d7b9916b29", - "label": "[Chargeback] Billing Components Overview", - "options": { - "openInNewTab": false, - "useCurrentDateRange": true, - "useCurrentFilters": true - }, - "order": 0, - "type": "dashboardLink" + "panelIndex": "eca9d216-830e-4509-88c7-66b1dd354c71", + "type": "visualization" + }, + { + "embeddableConfig": { + "attributes": { + "references": [], + "state": { + "adHocDataViews": { + "billing-realized-pool-dv": { + "allowHidden": false, + "allowNoIndex": false, + "fieldFormats": {}, + "id": "billing-realized-pool-dv", + "managed": false, + "name": "billing_realized_pool_lookup", + "runtimeFieldMap": {}, + "sourceFilters": [], + "timeFieldName": "@timestamp", + "title": "billing_realized_pool_lookup", + "type": "esql" + } + }, + "datasourceStates": { + "textBased": { + "indexPatternRefs": [ + { + "id": "billing-realized-pool-dv", + "timeField": "@timestamp", + "title": "billing_realized_pool_lookup" + } + ], + "layers": { + "28174526-fb0d-4645-a2b7-f5101203fb4e": { + "columns": [ + { + "columnId": "@timestamp", + "customLabel": false, + "fieldName": "@timestamp", + "label": "@timestamp", + "meta": { + "esType": "date", + "type": "date" + } }, { - "destinationRefName": "link_dc802531-4211-c74b-d9db-b197b528bbc8_dashboard", - "id": "dc802531-4211-c74b-d9db-b197b528bbc8", - "label": "[Chargeback] Usage & Cost Allocation", - "options": { - "openInNewTab": false, - "useCurrentDateRange": true, - "useCurrentFilters": true - }, - "order": 1, - "type": "dashboardLink" + "columnId": "deployment", + "customLabel": false, + "fieldName": "deployment", + "label": "Deployment", + "meta": { + "esType": "keyword", + "type": "string" + } }, { - "destinationRefName": "link_4e06e36e-f5d7-981f-8554-57756d1c59b8_dashboard", - "id": "4e06e36e-f5d7-981f-8554-57756d1c59b8", - "label": "[Chargeback] Configuration", - "options": { - "openInNewTab": false, - "useCurrentDateRange": true, - "useCurrentFilters": true - }, - "order": 2, - "type": "dashboardLink" + "columnId": "Normalized cost", + "customLabel": false, + "fieldName": "Normalized cost", + "inMetricDimension": true, + "label": "Normalized cost", + "meta": { + "esType": "double", + "type": "number" + }, + "params": { + "format": { + "id": "number", + "params": { + "decimals": 2 + } + } + } } - ] - }, - "gridData": { - "h": 2, - "i": "13cd2a6a-ec89-4553-ae64-cbc347770aa5", - "w": 48, - "x": 0, - "y": 0 - }, - "panelIndex": "13cd2a6a-ec89-4553-ae64-cbc347770aa5", - "type": "links" - }, - { - "embeddableConfig": { - "enhancements": { - "dynamicActions": { - "events": [] + ], + "index": "2bf6c0d816ef0a2d56d03ede549c16c08c35db2cf02d78c12756a98a33f50e4f", + "query": { + "esql": "FROM billing_realized_pool_lookup\n| WHERE @timestamp IS NOT NULL AND deployment_name IS NOT NULL\n| EVAL ts = @timestamp, deployment = CASE(deployment_id IS NULL, deployment_name, CONCAT(deployment_name, \" (\", LEFT(deployment_id, 6), \")\"))\n| LOOKUP JOIN cluster_capacity_utilization_lookup ON composite_key\n| LOOKUP JOIN chargeback_conf_lookup ON ts >= conf_start_date AND ts <= conf_end_date\n| EVAL mem_w = COALESCE(TO_DOUBLE(conf_utilization_memory_weight), 70.0), disk_w = COALESCE(TO_DOUBLE(conf_utilization_storage_weight), 30.0)\n| EVAL heap_p = COALESCE(heap_used_pct_p95, 100) / 100, disk_p = COALESCE(disk_used_pct_p95, 100) / 100\n| EVAL util_score = (mem_w * heap_p + disk_w * disk_p) / (mem_w + disk_w), chargeable_pool = data_tier_capacity_ecu * COALESCE(util_score, 1.0), rate = COALESCE(conf_chargeable_unit_rate, conf_ecu_rate, 1.0)\n| STATS `Normalized cost` = SUM(chargeable_pool * rate) BY ts, deployment\n| RENAME ts AS `@timestamp`\n| SORT `@timestamp` ASC\n| LIMIT 5000" + }, + "timeField": "@timestamp" + } + } + } + }, + "filters": [], + "needsRefresh": false, + "query": { + "esql": "FROM billing_realized_pool_lookup\n| WHERE @timestamp IS NOT NULL AND deployment_name IS NOT NULL\n| EVAL ts = @timestamp, deployment = CASE(deployment_id IS NULL, deployment_name, CONCAT(deployment_name, \" (\", LEFT(deployment_id, 6), \")\"))\n| LOOKUP JOIN cluster_capacity_utilization_lookup ON composite_key\n| LOOKUP JOIN chargeback_conf_lookup ON ts >= conf_start_date AND ts <= conf_end_date\n| EVAL mem_w = COALESCE(TO_DOUBLE(conf_utilization_memory_weight), 70.0), disk_w = COALESCE(TO_DOUBLE(conf_utilization_storage_weight), 30.0)\n| EVAL heap_p = COALESCE(heap_used_pct_p95, 100) / 100, disk_p = COALESCE(disk_used_pct_p95, 100) / 100\n| EVAL util_score = (mem_w * heap_p + disk_w * disk_p) / (mem_w + disk_w), chargeable_pool = data_tier_capacity_ecu * COALESCE(util_score, 1.0), rate = COALESCE(conf_chargeable_unit_rate, conf_ecu_rate, 1.0)\n| STATS `Normalized cost` = SUM(chargeable_pool * rate) BY ts, deployment\n| RENAME ts AS `@timestamp`\n| SORT `@timestamp` ASC\n| LIMIT 5000" + }, + "visualization": { + "axisTitlesVisibilitySettings": { + "x": true, + "yLeft": true, + "yRight": true + }, + "fittingFunction": "Linear", + "gridlinesVisibilitySettings": { + "x": true, + "yLeft": true, + "yRight": true + }, + "labelsOrientation": { + "x": 0, + "yLeft": 0, + "yRight": 0 + }, + "layers": [ + { + "accessors": [ + "Normalized cost" + ], + "colorMapping": { + "assignments": [], + "colorMode": { + "type": "categorical" + }, + "paletteId": "default", + "specialAssignments": [ + { + "color": { + "type": "loop" + }, + "rules": [ + { + "type": "other" + } + ], + "touched": false } + ] }, - "savedVis": { - "data": { - "aggs": [], - "searchSource": { - "filter": [], - "query": { - "language": "kuery", - "query": "" - } - } - }, - "description": "", - "id": "", - "params": { - "fontSize": 12, - "markdown": "# Data Tiers / Utilization", - "openLinksInNewTab": false + "layerId": "28174526-fb0d-4645-a2b7-f5101203fb4e", + "layerType": "data", + "seriesType": "bar_stacked", + "splitAccessor": "deployment", + "splitAccessors": [ + "deployment" + ], + "xAccessor": "@timestamp" + } + ], + "legend": { + "isVisible": true, + "position": "right" + }, + "preferredSeriesType": "bar_stacked", + "tickLabelsVisibilitySettings": { + "x": true, + "yLeft": true, + "yRight": true + }, + "valueLabels": "hide" + } + }, + "title": "Realized Pool per Deployment", + "version": 2, + "visualizationType": "lnsXY" + }, + "description": "Allocatable pool: data-tier capacity SKUs plus on-prem nodes, discounted by p95 utilization and converted at the configured rate. It excludes non-allocatable SKUs (ML, Kibana, snapshots, data transfer), so it is lower than the full deployment bill above.", + "drilldowns": [], + "title": "Realized Pool per Deployment" + }, + "gridData": { + "h": 16, + "i": "0c666274-4041-44ca-8a1e-7b4affafeb5d", + "sectionId": "0fceb22c-1e73-45e4-b634-006f6dcc5f43", + "w": 39, + "x": 9, + "y": 3 + }, + "panelIndex": "0c666274-4041-44ca-8a1e-7b4affafeb5d", + "type": "lens" + }, + { + "embeddableConfig": { + "attributes": { + "references": [], + "state": { + "adHocDataViews": { + "cluster-tier-contribution-dv": { + "allowHidden": false, + "allowNoIndex": false, + "fieldFormats": {}, + "id": "cluster-tier-contribution-dv", + "managed": false, + "name": "cluster_tier_contribution_lookup", + "runtimeFieldMap": {}, + "sourceFilters": [], + "timeFieldName": "@timestamp", + "title": "cluster_tier_contribution_lookup", + "type": "esql" + } + }, + "datasourceStates": { + "textBased": { + "indexPatternRefs": [ + { + "id": "cluster-tier-contribution-dv", + "timeField": "@timestamp", + "title": "cluster_tier_contribution_lookup" + } + ], + "layers": { + "tier-breakdown-layer": { + "columns": [ + { + "columnId": "Data Tier", + "customLabel": false, + "fieldName": "Data Tier", + "label": "Data Tier", + "meta": { + "esType": "keyword", + "type": "string" + } }, - "title": "", - "type": "markdown", - "uiState": {} + { + "columnId": "Normalized cost", + "customLabel": false, + "fieldName": "Normalized cost", + "inMetricDimension": true, + "label": "Normalized cost", + "meta": { + "esType": "double", + "type": "number" + }, + "params": { + "format": { + "id": "number", + "params": { + "decimals": 2 + } + } + } + } + ], + "index": "tier-breakdown-dv", + "query": { + "esql": "FROM cluster_tier_contribution_lookup\n| WHERE composite_key IS NOT NULL AND tier IS NOT NULL AND tier != \"unknown\"\n| EVAL ts = @timestamp\n| LOOKUP JOIN billing_realized_pool_lookup ON composite_key\n| WHERE data_tier_capacity_ecu IS NOT NULL\n| LOOKUP JOIN cluster_capacity_utilization_lookup ON composite_key\n| LOOKUP JOIN cluster_deployment_contribution_lookup ON composite_key\n| LOOKUP JOIN chargeback_conf_lookup ON ts >= conf_start_date AND ts <= conf_end_date\n| EVAL deployment = CASE(deployment_id IS NULL, deployment_name, CONCAT(deployment_name, \" (\", LEFT(deployment_id, 6), \")\"))\n| EVAL mem_w = COALESCE(TO_DOUBLE(conf_utilization_memory_weight), 70.0), disk_w = COALESCE(TO_DOUBLE(conf_utilization_storage_weight), 30.0)\n| EVAL heap_p = COALESCE(heap_used_pct_p95, 100) / 100, disk_p = COALESCE(disk_used_pct_p95, 100) / 100\n| EVAL util_score = (mem_w * heap_p + disk_w * disk_p) / (mem_w + disk_w), chargeable_pool = data_tier_capacity_ecu * util_score, tier_label = CASE(tier IS NULL OR tier == \"unknown\", \"Unclassified\", tier)\n| EVAL idx_w = COALESCE(TO_DOUBLE(conf_indexing_weight), 20.0), qry_w = COALESCE(TO_DOUBLE(conf_query_weight), 20.0), sto_w = COALESCE(TO_DOUBLE(conf_storage_weight), 40.0), rate = COALESCE(conf_chargeable_unit_rate, conf_ecu_rate, 1.0)\n| EVAL indexing = CASE(deployment_sum_indexing_time > 0, TO_DOUBLE(tier_sum_indexing_time) / deployment_sum_indexing_time * chargeable_pool), querying = CASE(deployment_sum_query_time > 0, TO_DOUBLE(tier_sum_query_time) / deployment_sum_query_time * chargeable_pool), store = CASE(deployment_sum_store_size > 0, TO_DOUBLE(tier_sum_store_size) / deployment_sum_store_size * chargeable_pool), data_set = CASE(deployment_sum_data_set_store_size > 0, TO_DOUBLE(tier_sum_data_set_store_size) / deployment_sum_data_set_store_size * chargeable_pool)\n| EVAL storage = CASE(store == 0, data_set, store), total_weight_hot = sto_w + qry_w + idx_w, total_weight_cold = sto_w + qry_w\n| EVAL blended = CASE(tier_label == \"hot/content\", (storage * sto_w + querying * qry_w + indexing * idx_w) / total_weight_hot, (storage * sto_w + querying * qry_w) / total_weight_cold) * rate\n| STATS `Normalized cost` = SUM(blended) BY tier_label\n| WHERE tier_label IS NOT NULL\n| RENAME tier_label AS `Data Tier`\n| SORT `Normalized cost` DESC\n| LIMIT 50" + }, + "timeField": "@timestamp" } - }, - "gridData": { - "h": 3, - "i": "eca9d216-830e-4509-88c7-66b1dd354c71", - "sectionId": "0fceb22c-1e73-45e4-b634-006f6dcc5f43", - "w": 48, - "x": 0, - "y": 0 - }, - "panelIndex": "eca9d216-830e-4509-88c7-66b1dd354c71", - "type": "visualization" + } + } + }, + "filters": [], + "needsRefresh": false, + "query": { + "esql": "FROM cluster_tier_contribution_lookup\n| WHERE composite_key IS NOT NULL AND tier IS NOT NULL AND tier != \"unknown\"\n| EVAL ts = @timestamp\n| LOOKUP JOIN billing_realized_pool_lookup ON composite_key\n| WHERE data_tier_capacity_ecu IS NOT NULL\n| LOOKUP JOIN cluster_capacity_utilization_lookup ON composite_key\n| LOOKUP JOIN cluster_deployment_contribution_lookup ON composite_key\n| LOOKUP JOIN chargeback_conf_lookup ON ts >= conf_start_date AND ts <= conf_end_date\n| EVAL deployment = CASE(deployment_id IS NULL, deployment_name, CONCAT(deployment_name, \" (\", LEFT(deployment_id, 6), \")\"))\n| EVAL mem_w = COALESCE(TO_DOUBLE(conf_utilization_memory_weight), 70.0), disk_w = COALESCE(TO_DOUBLE(conf_utilization_storage_weight), 30.0)\n| EVAL heap_p = COALESCE(heap_used_pct_p95, 100) / 100, disk_p = COALESCE(disk_used_pct_p95, 100) / 100\n| EVAL util_score = (mem_w * heap_p + disk_w * disk_p) / (mem_w + disk_w), chargeable_pool = data_tier_capacity_ecu * util_score, tier_label = CASE(tier IS NULL OR tier == \"unknown\", \"Unclassified\", tier)\n| EVAL idx_w = COALESCE(TO_DOUBLE(conf_indexing_weight), 20.0), qry_w = COALESCE(TO_DOUBLE(conf_query_weight), 20.0), sto_w = COALESCE(TO_DOUBLE(conf_storage_weight), 40.0), rate = COALESCE(conf_chargeable_unit_rate, conf_ecu_rate, 1.0)\n| EVAL indexing = CASE(deployment_sum_indexing_time > 0, TO_DOUBLE(tier_sum_indexing_time) / deployment_sum_indexing_time * chargeable_pool), querying = CASE(deployment_sum_query_time > 0, TO_DOUBLE(tier_sum_query_time) / deployment_sum_query_time * chargeable_pool), store = CASE(deployment_sum_store_size > 0, TO_DOUBLE(tier_sum_store_size) / deployment_sum_store_size * chargeable_pool), data_set = CASE(deployment_sum_data_set_store_size > 0, TO_DOUBLE(tier_sum_data_set_store_size) / deployment_sum_data_set_store_size * chargeable_pool)\n| EVAL storage = CASE(store == 0, data_set, store), total_weight_hot = sto_w + qry_w + idx_w, total_weight_cold = sto_w + qry_w\n| EVAL blended = CASE(tier_label == \"hot/content\", (storage * sto_w + querying * qry_w + indexing * idx_w) / total_weight_hot, (storage * sto_w + querying * qry_w) / total_weight_cold) * rate\n| STATS `Normalized cost` = SUM(blended) BY tier_label\n| WHERE tier_label IS NOT NULL\n| RENAME tier_label AS `Data Tier`\n| SORT `Normalized cost` DESC\n| LIMIT 50" + }, + "visualization": { + "columns": [ + { + "columnId": "Data Tier", + "isMetric": false, + "isTransposed": false + }, + { + "columnId": "Normalized cost", + "isMetric": true, + "isTransposed": false, + "summaryRow": "sum" + } + ], + "layerId": "tier-breakdown-layer", + "layerType": "data" + } }, - { - "embeddableConfig": { - "attributes": { - "references": [], - "state": { - "adHocDataViews": { - "billing-realized-pool-dv": { - "allowHidden": false, - "allowNoIndex": false, - "fieldFormats": {}, - "id": "billing-realized-pool-dv", - "managed": false, - "name": "billing_realized_pool_lookup", - "runtimeFieldMap": {}, - "sourceFilters": [], - "timeFieldName": "@timestamp", - "title": "billing_realized_pool_lookup", - "type": "esql" - } - }, - "datasourceStates": { - "textBased": { - "indexPatternRefs": [ - { - "id": "billing-realized-pool-dv", - "timeField": "@timestamp", - "title": "billing_realized_pool_lookup" - } - ], - "layers": { - "28174526-fb0d-4645-a2b7-f5101203fb4e": { - "columns": [ - { - "columnId": "@timestamp", - "customLabel": false, - "fieldName": "@timestamp", - "label": "@timestamp", - "meta": { - "esType": "date", - "type": "date" - } - }, - { - "columnId": "deployment", - "customLabel": false, - "fieldName": "deployment", - "label": "Deployment", - "meta": { - "esType": "keyword", - "type": "string" - } - }, - { - "columnId": "Normalized cost", - "customLabel": false, - "fieldName": "Normalized cost", - "inMetricDimension": true, - "label": "Normalized cost", - "meta": { - "esType": "double", - "type": "number" - }, - "params": { - "format": { - "id": "number", - "params": { - "decimals": 2 - } - } - } - } - ], - "index": "2bf6c0d816ef0a2d56d03ede549c16c08c35db2cf02d78c12756a98a33f50e4f", - "query": { - "esql": "FROM billing_realized_pool_lookup\n| WHERE @timestamp IS NOT NULL AND deployment_name IS NOT NULL\n| EVAL ts = @timestamp, deployment = CASE(deployment_id IS NULL, deployment_name, CONCAT(deployment_name, \" (\", LEFT(deployment_id, 6), \")\"))\n| LOOKUP JOIN cluster_capacity_utilization_lookup ON composite_key\n| LOOKUP JOIN chargeback_conf_lookup ON ts >= conf_start_date AND ts <= conf_end_date\n| EVAL mem_w = COALESCE(TO_DOUBLE(conf_utilization_memory_weight), 70.0), disk_w = COALESCE(TO_DOUBLE(conf_utilization_storage_weight), 30.0)\n| EVAL heap_p = COALESCE(heap_used_pct_p95, 100) / 100, disk_p = COALESCE(disk_used_pct_p95, 100) / 100\n| EVAL util_score = (mem_w * heap_p + disk_w * disk_p) / (mem_w + disk_w), chargeable_pool = data_tier_capacity_ecu * COALESCE(util_score, 1.0), rate = COALESCE(conf_chargeable_unit_rate, conf_ecu_rate, 1.0)\n| STATS `Normalized cost` = SUM(chargeable_pool * rate) BY ts, deployment\n| RENAME ts AS `@timestamp`\n| SORT `@timestamp` ASC\n| LIMIT 5000" - }, - "timeField": "@timestamp" - } - } - } - }, - "filters": [], - "needsRefresh": false, - "query": { - "esql": "FROM billing_realized_pool_lookup\n| WHERE @timestamp IS NOT NULL AND deployment_name IS NOT NULL\n| EVAL ts = @timestamp, deployment = CASE(deployment_id IS NULL, deployment_name, CONCAT(deployment_name, \" (\", LEFT(deployment_id, 6), \")\"))\n| LOOKUP JOIN cluster_capacity_utilization_lookup ON composite_key\n| LOOKUP JOIN chargeback_conf_lookup ON ts >= conf_start_date AND ts <= conf_end_date\n| EVAL mem_w = COALESCE(TO_DOUBLE(conf_utilization_memory_weight), 70.0), disk_w = COALESCE(TO_DOUBLE(conf_utilization_storage_weight), 30.0)\n| EVAL heap_p = COALESCE(heap_used_pct_p95, 100) / 100, disk_p = COALESCE(disk_used_pct_p95, 100) / 100\n| EVAL util_score = (mem_w * heap_p + disk_w * disk_p) / (mem_w + disk_w), chargeable_pool = data_tier_capacity_ecu * COALESCE(util_score, 1.0), rate = COALESCE(conf_chargeable_unit_rate, conf_ecu_rate, 1.0)\n| STATS `Normalized cost` = SUM(chargeable_pool * rate) BY ts, deployment\n| RENAME ts AS `@timestamp`\n| SORT `@timestamp` ASC\n| LIMIT 5000" - }, - "visualization": { - "axisTitlesVisibilitySettings": { - "x": true, - "yLeft": true, - "yRight": true - }, - "fittingFunction": "Linear", - "gridlinesVisibilitySettings": { - "x": true, - "yLeft": true, - "yRight": true - }, - "labelsOrientation": { - "x": 0, - "yLeft": 0, - "yRight": 0 - }, - "layers": [ - { - "accessors": [ - "Normalized cost" - ], - "colorMapping": { - "assignments": [], - "colorMode": { - "type": "categorical" - }, - "paletteId": "default", - "specialAssignments": [ - { - "color": { - "type": "loop" - }, - "rules": [ - { - "type": "other" - } - ], - "touched": false - } - ] - }, - "layerId": "28174526-fb0d-4645-a2b7-f5101203fb4e", - "layerType": "data", - "seriesType": "bar_stacked", - "splitAccessor": "deployment", - "splitAccessors": [ - "deployment" - ], - "xAccessor": "@timestamp" - } - ], - "legend": { - "isVisible": true, - "position": "right" - }, - "preferredSeriesType": "bar_stacked", - "tickLabelsVisibilitySettings": { - "x": true, - "yLeft": true, - "yRight": true - }, - "valueLabels": "hide" + "title": "Chargeable Pool by Data Tier", + "version": 2, + "visualization": { + "layerId": "tier-breakdown-dv", + "layerType": "data" + }, + "visualizationType": "lnsDatatable" + }, + "description": "Full deployment bill: the sum of ALL billing SKUs (data tiers, ML, Kibana, snapshots, data transfer and on-prem nodes) converted at the configured rate. This is the highest-level total. Tier and data-stream panels show only the allocatable data-tier pool, so they sum to less than this figure.", + "drilldowns": [], + "title": "Chargeable Pool by Data Tier" + }, + "gridData": { + "h": 16, + "i": "bb5dac10-896d-4312-a0db-4aa663d0d88b", + "sectionId": "0fceb22c-1e73-45e4-b634-006f6dcc5f43", + "w": 9, + "x": 0, + "y": 3 + }, + "panelIndex": "bb5dac10-896d-4312-a0db-4aa663d0d88b", + "type": "lens" + }, + { + "embeddableConfig": { + "attributes": { + "references": [], + "state": { + "adHocDataViews": { + "billing-realized-pool-dv": { + "allowHidden": false, + "allowNoIndex": false, + "fieldFormats": {}, + "id": "billing-realized-pool-dv", + "managed": false, + "name": "billing_realized_pool_lookup", + "runtimeFieldMap": {}, + "sourceFilters": [], + "timeFieldName": "@timestamp", + "title": "billing_realized_pool_lookup", + "type": "esql" + } + }, + "datasourceStates": { + "textBased": { + "indexPatternRefs": [ + { + "id": "billing-realized-pool-dv", + "timeField": "@timestamp", + "title": "billing_realized_pool_lookup" + } + ], + "layers": { + "9d199172-f9fa-4fc4-82af-c7e2672a3f54": { + "columns": [ + { + "columnId": "Data Tier", + "customLabel": false, + "fieldName": "Data Tier", + "label": "Data Tier", + "meta": { + "esType": "keyword", + "type": "string" + } + }, + { + "columnId": "Indexing", + "customLabel": false, + "fieldName": "Indexing", + "inMetricDimension": true, + "label": "Indexing", + "meta": { + "esType": "double", + "type": "number" + }, + "params": { + "format": { + "id": "number", + "params": { + "decimals": 2 + } } + } }, - "title": "Realized Pool per Deployment", - "version": 2, - "visualizationType": "lnsXY" - }, - "description": "Allocatable pool: data-tier capacity SKUs plus on-prem nodes, discounted by p95 utilization and converted at the configured rate. It excludes non-allocatable SKUs (ML, Kibana, snapshots, data transfer), so it is lower than the full deployment bill above.", - "drilldowns": [], - "title": "Realized Pool per Deployment" - }, - "gridData": { - "h": 16, - "i": "0c666274-4041-44ca-8a1e-7b4affafeb5d", - "sectionId": "0fceb22c-1e73-45e4-b634-006f6dcc5f43", - "w": 39, - "x": 9, - "y": 3 - }, - "panelIndex": "0c666274-4041-44ca-8a1e-7b4affafeb5d", - "type": "lens" - }, - { - "embeddableConfig": { - "attributes": { - "references": [], - "state": { - "adHocDataViews": { - "cluster-tier-contribution-dv": { - "allowHidden": false, - "allowNoIndex": false, - "fieldFormats": {}, - "id": "cluster-tier-contribution-dv", - "managed": false, - "name": "cluster_tier_contribution_lookup", - "runtimeFieldMap": {}, - "sourceFilters": [], - "timeFieldName": "@timestamp", - "title": "cluster_tier_contribution_lookup", - "type": "esql" - } - }, - "datasourceStates": { - "textBased": { - "indexPatternRefs": [ - { - "id": "cluster-tier-contribution-dv", - "timeField": "@timestamp", - "title": "cluster_tier_contribution_lookup" - } - ], - "layers": { - "tier-breakdown-layer": { - "columns": [ - { - "columnId": "Data Tier", - "customLabel": false, - "fieldName": "Data Tier", - "label": "Data Tier", - "meta": { - "esType": "keyword", - "type": "string" - } - }, - { - "columnId": "Normalized cost", - "customLabel": false, - "fieldName": "Normalized cost", - "inMetricDimension": true, - "label": "Normalized cost", - "meta": { - "esType": "double", - "type": "number" - }, - "params": { - "format": { - "id": "number", - "params": { - "decimals": 2 - } - } - } - } - ], - "index": "tier-breakdown-dv", - "query": { - "esql": "FROM cluster_tier_contribution_lookup\n| WHERE composite_key IS NOT NULL AND tier IS NOT NULL AND tier != \"unknown\"\n| EVAL ts = @timestamp\n| LOOKUP JOIN billing_realized_pool_lookup ON composite_key\n| WHERE data_tier_capacity_ecu IS NOT NULL\n| LOOKUP JOIN cluster_capacity_utilization_lookup ON composite_key\n| LOOKUP JOIN cluster_deployment_contribution_lookup ON composite_key\n| LOOKUP JOIN chargeback_conf_lookup ON ts >= conf_start_date AND ts <= conf_end_date\n| EVAL deployment = CASE(deployment_id IS NULL, deployment_name, CONCAT(deployment_name, \" (\", LEFT(deployment_id, 6), \")\"))\n| EVAL mem_w = COALESCE(TO_DOUBLE(conf_utilization_memory_weight), 70.0), disk_w = COALESCE(TO_DOUBLE(conf_utilization_storage_weight), 30.0)\n| EVAL heap_p = COALESCE(heap_used_pct_p95, 100) / 100, disk_p = COALESCE(disk_used_pct_p95, 100) / 100\n| EVAL util_score = (mem_w * heap_p + disk_w * disk_p) / (mem_w + disk_w), chargeable_pool = data_tier_capacity_ecu * util_score, tier_label = CASE(tier IS NULL OR tier == \"unknown\", \"Unclassified\", tier)\n| EVAL idx_w = COALESCE(TO_DOUBLE(conf_indexing_weight), 20.0), qry_w = COALESCE(TO_DOUBLE(conf_query_weight), 20.0), sto_w = COALESCE(TO_DOUBLE(conf_storage_weight), 40.0), rate = COALESCE(conf_chargeable_unit_rate, conf_ecu_rate, 1.0)\n| EVAL indexing = CASE(deployment_sum_indexing_time > 0, TO_DOUBLE(tier_sum_indexing_time) / deployment_sum_indexing_time * chargeable_pool), querying = CASE(deployment_sum_query_time > 0, TO_DOUBLE(tier_sum_query_time) / deployment_sum_query_time * chargeable_pool), store = CASE(deployment_sum_store_size > 0, TO_DOUBLE(tier_sum_store_size) / deployment_sum_store_size * chargeable_pool), data_set = CASE(deployment_sum_data_set_store_size > 0, TO_DOUBLE(tier_sum_data_set_store_size) / deployment_sum_data_set_store_size * chargeable_pool)\n| EVAL storage = CASE(store == 0, data_set, store), total_weight_hot = sto_w + qry_w + idx_w, total_weight_cold = sto_w + qry_w\n| EVAL blended = CASE(tier_label == \"hot/content\", (storage * sto_w + querying * qry_w + indexing * idx_w) / total_weight_hot, (storage * sto_w + querying * qry_w) / total_weight_cold) * rate\n| STATS `Normalized cost` = SUM(blended) BY tier_label\n| WHERE tier_label IS NOT NULL\n| RENAME tier_label AS `Data Tier`\n| SORT `Normalized cost` DESC\n| LIMIT 50" - }, - "timeField": "@timestamp" - } - } - } - }, - "filters": [], - "needsRefresh": false, - "query": { - "esql": "FROM cluster_tier_contribution_lookup\n| WHERE composite_key IS NOT NULL AND tier IS NOT NULL AND tier != \"unknown\"\n| EVAL ts = @timestamp\n| LOOKUP JOIN billing_realized_pool_lookup ON composite_key\n| WHERE data_tier_capacity_ecu IS NOT NULL\n| LOOKUP JOIN cluster_capacity_utilization_lookup ON composite_key\n| LOOKUP JOIN cluster_deployment_contribution_lookup ON composite_key\n| LOOKUP JOIN chargeback_conf_lookup ON ts >= conf_start_date AND ts <= conf_end_date\n| EVAL deployment = CASE(deployment_id IS NULL, deployment_name, CONCAT(deployment_name, \" (\", LEFT(deployment_id, 6), \")\"))\n| EVAL mem_w = COALESCE(TO_DOUBLE(conf_utilization_memory_weight), 70.0), disk_w = COALESCE(TO_DOUBLE(conf_utilization_storage_weight), 30.0)\n| EVAL heap_p = COALESCE(heap_used_pct_p95, 100) / 100, disk_p = COALESCE(disk_used_pct_p95, 100) / 100\n| EVAL util_score = (mem_w * heap_p + disk_w * disk_p) / (mem_w + disk_w), chargeable_pool = data_tier_capacity_ecu * util_score, tier_label = CASE(tier IS NULL OR tier == \"unknown\", \"Unclassified\", tier)\n| EVAL idx_w = COALESCE(TO_DOUBLE(conf_indexing_weight), 20.0), qry_w = COALESCE(TO_DOUBLE(conf_query_weight), 20.0), sto_w = COALESCE(TO_DOUBLE(conf_storage_weight), 40.0), rate = COALESCE(conf_chargeable_unit_rate, conf_ecu_rate, 1.0)\n| EVAL indexing = CASE(deployment_sum_indexing_time > 0, TO_DOUBLE(tier_sum_indexing_time) / deployment_sum_indexing_time * chargeable_pool), querying = CASE(deployment_sum_query_time > 0, TO_DOUBLE(tier_sum_query_time) / deployment_sum_query_time * chargeable_pool), store = CASE(deployment_sum_store_size > 0, TO_DOUBLE(tier_sum_store_size) / deployment_sum_store_size * chargeable_pool), data_set = CASE(deployment_sum_data_set_store_size > 0, TO_DOUBLE(tier_sum_data_set_store_size) / deployment_sum_data_set_store_size * chargeable_pool)\n| EVAL storage = CASE(store == 0, data_set, store), total_weight_hot = sto_w + qry_w + idx_w, total_weight_cold = sto_w + qry_w\n| EVAL blended = CASE(tier_label == \"hot/content\", (storage * sto_w + querying * qry_w + indexing * idx_w) / total_weight_hot, (storage * sto_w + querying * qry_w) / total_weight_cold) * rate\n| STATS `Normalized cost` = SUM(blended) BY tier_label\n| WHERE tier_label IS NOT NULL\n| RENAME tier_label AS `Data Tier`\n| SORT `Normalized cost` DESC\n| LIMIT 50" - }, - "visualization": { - "columns": [ - { - "columnId": "Data Tier", - "isMetric": false, - "isTransposed": false - }, - { - "columnId": "Normalized cost", - "isMetric": true, - "isTransposed": false, - "summaryRow": "sum" - } - ], - "layerId": "tier-breakdown-layer", - "layerType": "data" + { + "columnId": "Querying", + "customLabel": false, + "fieldName": "Querying", + "inMetricDimension": true, + "label": "Querying", + "meta": { + "esType": "double", + "type": "number" + }, + "params": { + "format": { + "id": "number", + "params": { + "decimals": 2 + } } + } }, - "title": "Chargeable Pool by Data Tier", - "version": 2, - "visualization": { - "layerId": "tier-breakdown-dv", - "layerType": "data" + { + "columnId": "Storage", + "customLabel": false, + "fieldName": "Storage", + "inMetricDimension": true, + "label": "Storage", + "meta": { + "esType": "double", + "type": "number" + }, + "params": { + "format": { + "id": "number", + "params": { + "decimals": 2 + } + } + } }, - "visualizationType": "lnsDatatable" - }, - "description": "Full deployment bill: the sum of ALL billing SKUs (data tiers, ML, Kibana, snapshots, data transfer and on-prem nodes) converted at the configured rate. This is the highest-level total. Tier and data-stream panels show only the allocatable data-tier pool, so they sum to less than this figure.", - "drilldowns": [], - "title": "Chargeable Pool by Data Tier" - }, - "gridData": { - "h": 16, - "i": "bb5dac10-896d-4312-a0db-4aa663d0d88b", - "sectionId": "0fceb22c-1e73-45e4-b634-006f6dcc5f43", - "w": 9, - "x": 0, - "y": 3 - }, - "panelIndex": "bb5dac10-896d-4312-a0db-4aa663d0d88b", - "type": "lens" + { + "columnId": "Blended", + "customLabel": false, + "fieldName": "Blended", + "inMetricDimension": true, + "label": "Blended", + "meta": { + "esType": "double", + "type": "number" + }, + "params": { + "format": { + "id": "number", + "params": { + "decimals": 2 + } + } + } + } + ], + "index": "chargeback_integration", + "query": { + "esql": "FROM billing_realized_pool_lookup\n| LOOKUP JOIN cluster_capacity_utilization_lookup ON composite_key\n| LOOKUP JOIN cluster_deployment_contribution_lookup ON composite_key\n| LOOKUP JOIN cluster_tier_contribution_lookup ON composite_key\n| LOOKUP JOIN chargeback_conf_lookup ON @timestamp >= conf_start_date AND @timestamp <= conf_end_date\n| EVAL heap_util = COALESCE(heap_used_pct_p95, 100) / 100, disk_util = COALESCE(disk_used_pct_p95, 100) / 100, w_sum = conf_utilization_memory_weight + conf_utilization_storage_weight, capacity_utilization_score = CASE(w_sum > 0, (conf_utilization_memory_weight * heap_util + conf_utilization_storage_weight * disk_util) / w_sum, 1.0)\n| EVAL chargeable_pool = data_tier_capacity_ecu * COALESCE(capacity_utilization_score, 1.0), tier_label = CASE(tier IS NULL OR tier == \"unknown\", \"Unclassified\", tier)\n| EVAL indexing = CASE (deployment_sum_indexing_time > 0, TO_DOUBLE(tier_sum_indexing_time) / deployment_sum_indexing_time * chargeable_pool) * COALESCE(conf_chargeable_unit_rate, conf_ecu_rate), querying = CASE (deployment_sum_query_time > 0, TO_DOUBLE(tier_sum_query_time) / deployment_sum_query_time * chargeable_pool) * COALESCE(conf_chargeable_unit_rate, conf_ecu_rate), data_set = CASE (deployment_sum_data_set_store_size > 0, TO_DOUBLE(tier_sum_data_set_store_size) / deployment_sum_data_set_store_size * chargeable_pool), store = CASE (deployment_sum_store_size > 0, TO_DOUBLE(tier_sum_store_size) / deployment_sum_store_size * chargeable_pool), storage = CASE (store == 0, data_set, store) * COALESCE(conf_chargeable_unit_rate, conf_ecu_rate), total_weight_hot = conf_storage_weight + conf_query_weight + conf_indexing_weight, total_weight_cold = conf_storage_weight + conf_query_weight, blended = CASE (tier_label == \"hot/content\", ((storage / COALESCE(conf_chargeable_unit_rate, conf_ecu_rate) * conf_storage_weight) + (querying / COALESCE(conf_chargeable_unit_rate, conf_ecu_rate) * conf_query_weight) + (indexing / COALESCE(conf_chargeable_unit_rate, conf_ecu_rate) * conf_indexing_weight)) / total_weight_hot, ((storage / COALESCE(conf_chargeable_unit_rate, conf_ecu_rate) * conf_storage_weight) + (querying / COALESCE(conf_chargeable_unit_rate, conf_ecu_rate) * conf_query_weight)) / total_weight_cold) * COALESCE(conf_chargeable_unit_rate, conf_ecu_rate)\n| STATS Indexing = SUM(indexing), Querying = SUM(querying), Storage = SUM(storage), Blended = SUM(blended) BY tier_label\n| WHERE Blended > 0\n| RENAME tier_label AS `Data Tier`\n| SORT Blended DESC" + }, + "timeField": "@timestamp" + } + } + } + }, + "filters": [], + "needsRefresh": false, + "query": { + "esql": "FROM billing_realized_pool_lookup\n| LOOKUP JOIN cluster_capacity_utilization_lookup ON composite_key\n| LOOKUP JOIN cluster_deployment_contribution_lookup ON composite_key\n| LOOKUP JOIN cluster_tier_contribution_lookup ON composite_key\n| LOOKUP JOIN chargeback_conf_lookup ON @timestamp >= conf_start_date AND @timestamp <= conf_end_date\n| EVAL heap_util = COALESCE(heap_used_pct_p95, 100) / 100, disk_util = COALESCE(disk_used_pct_p95, 100) / 100, w_sum = conf_utilization_memory_weight + conf_utilization_storage_weight, capacity_utilization_score = CASE(w_sum > 0, (conf_utilization_memory_weight * heap_util + conf_utilization_storage_weight * disk_util) / w_sum, 1.0)\n| EVAL chargeable_pool = data_tier_capacity_ecu * COALESCE(capacity_utilization_score, 1.0), tier_label = CASE(tier IS NULL OR tier == \"unknown\", \"Unclassified\", tier)\n| EVAL indexing = CASE (deployment_sum_indexing_time > 0, TO_DOUBLE(tier_sum_indexing_time) / deployment_sum_indexing_time * chargeable_pool) * COALESCE(conf_chargeable_unit_rate, conf_ecu_rate), querying = CASE (deployment_sum_query_time > 0, TO_DOUBLE(tier_sum_query_time) / deployment_sum_query_time * chargeable_pool) * COALESCE(conf_chargeable_unit_rate, conf_ecu_rate), data_set = CASE (deployment_sum_data_set_store_size > 0, TO_DOUBLE(tier_sum_data_set_store_size) / deployment_sum_data_set_store_size * chargeable_pool), store = CASE (deployment_sum_store_size > 0, TO_DOUBLE(tier_sum_store_size) / deployment_sum_store_size * chargeable_pool), storage = CASE (store == 0, data_set, store) * COALESCE(conf_chargeable_unit_rate, conf_ecu_rate), total_weight_hot = conf_storage_weight + conf_query_weight + conf_indexing_weight, total_weight_cold = conf_storage_weight + conf_query_weight, blended = CASE (tier_label == \"hot/content\", ((storage / COALESCE(conf_chargeable_unit_rate, conf_ecu_rate) * conf_storage_weight) + (querying / COALESCE(conf_chargeable_unit_rate, conf_ecu_rate) * conf_query_weight) + (indexing / COALESCE(conf_chargeable_unit_rate, conf_ecu_rate) * conf_indexing_weight)) / total_weight_hot, ((storage / COALESCE(conf_chargeable_unit_rate, conf_ecu_rate) * conf_storage_weight) + (querying / COALESCE(conf_chargeable_unit_rate, conf_ecu_rate) * conf_query_weight)) / total_weight_cold) * COALESCE(conf_chargeable_unit_rate, conf_ecu_rate)\n| STATS Indexing = SUM(indexing), Querying = SUM(querying), Storage = SUM(storage), Blended = SUM(blended) BY tier_label\n| WHERE Blended > 0\n| RENAME tier_label AS `Data Tier`\n| SORT Blended DESC" + }, + "visualization": { + "columns": [ + { + "columnId": "Data Tier", + "isMetric": false, + "isTransposed": false + }, + { + "columnId": "Indexing", + "isMetric": true, + "isTransposed": false, + "summaryRow": "sum" + }, + { + "columnId": "Querying", + "isMetric": true, + "isTransposed": false, + "summaryRow": "sum" + }, + { + "columnId": "Storage", + "isMetric": true, + "isTransposed": false, + "summaryRow": "sum" + }, + { + "columnId": "Blended", + "isMetric": true, + "isTransposed": false, + "summaryRow": "sum" + } + ], + "layerId": "9d199172-f9fa-4fc4-82af-c7e2672a3f54", + "layerType": "data" + } }, - { - "embeddableConfig": { - "attributes": { - "references": [], - "state": { - "adHocDataViews": { - "billing-realized-pool-dv": { - "allowHidden": false, - "allowNoIndex": false, - "fieldFormats": {}, - "id": "billing-realized-pool-dv", - "managed": false, - "name": "billing_realized_pool_lookup", - "runtimeFieldMap": {}, - "sourceFilters": [], - "timeFieldName": "@timestamp", - "title": "billing_realized_pool_lookup", - "type": "esql" - } - }, - "datasourceStates": { - "textBased": { - "indexPatternRefs": [ - { - "id": "billing-realized-pool-dv", - "timeField": "@timestamp", - "title": "billing_realized_pool_lookup" - } - ], - "layers": { - "9d199172-f9fa-4fc4-82af-c7e2672a3f54": { - "columns": [ - { - "columnId": "Data Tier", - "customLabel": false, - "fieldName": "Data Tier", - "label": "Data Tier", - "meta": { - "esType": "keyword", - "type": "string" - } - }, - { - "columnId": "Indexing", - "customLabel": false, - "fieldName": "Indexing", - "inMetricDimension": true, - "label": "Indexing", - "meta": { - "esType": "double", - "type": "number" - }, - "params": { - "format": { - "id": "number", - "params": { - "decimals": 2 - } - } - } - }, - { - "columnId": "Querying", - "customLabel": false, - "fieldName": "Querying", - "inMetricDimension": true, - "label": "Querying", - "meta": { - "esType": "double", - "type": "number" - }, - "params": { - "format": { - "id": "number", - "params": { - "decimals": 2 - } - } - } - }, - { - "columnId": "Storage", - "customLabel": false, - "fieldName": "Storage", - "inMetricDimension": true, - "label": "Storage", - "meta": { - "esType": "double", - "type": "number" - }, - "params": { - "format": { - "id": "number", - "params": { - "decimals": 2 - } - } - } - }, - { - "columnId": "Blended", - "customLabel": false, - "fieldName": "Blended", - "inMetricDimension": true, - "label": "Blended", - "meta": { - "esType": "double", - "type": "number" - }, - "params": { - "format": { - "id": "number", - "params": { - "decimals": 2 - } - } - } - } - ], - "index": "chargeback_integration", - "query": { - "esql": "FROM billing_realized_pool_lookup\n| LOOKUP JOIN cluster_capacity_utilization_lookup ON composite_key\n| LOOKUP JOIN cluster_deployment_contribution_lookup ON composite_key\n| LOOKUP JOIN cluster_tier_contribution_lookup ON composite_key\n| LOOKUP JOIN chargeback_conf_lookup ON @timestamp >= conf_start_date AND @timestamp <= conf_end_date\n| EVAL heap_util = COALESCE(heap_used_pct_p95, 100) / 100, disk_util = COALESCE(disk_used_pct_p95, 100) / 100, w_sum = conf_utilization_memory_weight + conf_utilization_storage_weight, capacity_utilization_score = CASE(w_sum > 0, (conf_utilization_memory_weight * heap_util + conf_utilization_storage_weight * disk_util) / w_sum, 1.0)\n| EVAL chargeable_pool = data_tier_capacity_ecu * COALESCE(capacity_utilization_score, 1.0), tier_label = CASE(tier IS NULL OR tier == \"unknown\", \"Unclassified\", tier)\n| EVAL indexing = CASE (deployment_sum_indexing_time > 0, TO_DOUBLE(tier_sum_indexing_time) / deployment_sum_indexing_time * chargeable_pool) * COALESCE(conf_chargeable_unit_rate, conf_ecu_rate), querying = CASE (deployment_sum_query_time > 0, TO_DOUBLE(tier_sum_query_time) / deployment_sum_query_time * chargeable_pool) * COALESCE(conf_chargeable_unit_rate, conf_ecu_rate), data_set = CASE (deployment_sum_data_set_store_size > 0, TO_DOUBLE(tier_sum_data_set_store_size) / deployment_sum_data_set_store_size * chargeable_pool), store = CASE (deployment_sum_store_size > 0, TO_DOUBLE(tier_sum_store_size) / deployment_sum_store_size * chargeable_pool), storage = CASE (store == 0, data_set, store) * COALESCE(conf_chargeable_unit_rate, conf_ecu_rate), total_weight_hot = conf_storage_weight + conf_query_weight + conf_indexing_weight, total_weight_cold = conf_storage_weight + conf_query_weight, blended = CASE (tier_label == \"hot/content\", ((storage / COALESCE(conf_chargeable_unit_rate, conf_ecu_rate) * conf_storage_weight) + (querying / COALESCE(conf_chargeable_unit_rate, conf_ecu_rate) * conf_query_weight) + (indexing / COALESCE(conf_chargeable_unit_rate, conf_ecu_rate) * conf_indexing_weight)) / total_weight_hot, ((storage / COALESCE(conf_chargeable_unit_rate, conf_ecu_rate) * conf_storage_weight) + (querying / COALESCE(conf_chargeable_unit_rate, conf_ecu_rate) * conf_query_weight)) / total_weight_cold) * COALESCE(conf_chargeable_unit_rate, conf_ecu_rate)\n| STATS Indexing = SUM(indexing), Querying = SUM(querying), Storage = SUM(storage), Blended = SUM(blended) BY tier_label\n| WHERE Blended > 0\n| RENAME tier_label AS `Data Tier`\n| SORT Blended DESC" - }, - "timeField": "@timestamp" - } - } - } - }, - "filters": [], - "needsRefresh": false, - "query": { - "esql": "FROM billing_realized_pool_lookup\n| LOOKUP JOIN cluster_capacity_utilization_lookup ON composite_key\n| LOOKUP JOIN cluster_deployment_contribution_lookup ON composite_key\n| LOOKUP JOIN cluster_tier_contribution_lookup ON composite_key\n| LOOKUP JOIN chargeback_conf_lookup ON @timestamp >= conf_start_date AND @timestamp <= conf_end_date\n| EVAL heap_util = COALESCE(heap_used_pct_p95, 100) / 100, disk_util = COALESCE(disk_used_pct_p95, 100) / 100, w_sum = conf_utilization_memory_weight + conf_utilization_storage_weight, capacity_utilization_score = CASE(w_sum > 0, (conf_utilization_memory_weight * heap_util + conf_utilization_storage_weight * disk_util) / w_sum, 1.0)\n| EVAL chargeable_pool = data_tier_capacity_ecu * COALESCE(capacity_utilization_score, 1.0), tier_label = CASE(tier IS NULL OR tier == \"unknown\", \"Unclassified\", tier)\n| EVAL indexing = CASE (deployment_sum_indexing_time > 0, TO_DOUBLE(tier_sum_indexing_time) / deployment_sum_indexing_time * chargeable_pool) * COALESCE(conf_chargeable_unit_rate, conf_ecu_rate), querying = CASE (deployment_sum_query_time > 0, TO_DOUBLE(tier_sum_query_time) / deployment_sum_query_time * chargeable_pool) * COALESCE(conf_chargeable_unit_rate, conf_ecu_rate), data_set = CASE (deployment_sum_data_set_store_size > 0, TO_DOUBLE(tier_sum_data_set_store_size) / deployment_sum_data_set_store_size * chargeable_pool), store = CASE (deployment_sum_store_size > 0, TO_DOUBLE(tier_sum_store_size) / deployment_sum_store_size * chargeable_pool), storage = CASE (store == 0, data_set, store) * COALESCE(conf_chargeable_unit_rate, conf_ecu_rate), total_weight_hot = conf_storage_weight + conf_query_weight + conf_indexing_weight, total_weight_cold = conf_storage_weight + conf_query_weight, blended = CASE (tier_label == \"hot/content\", ((storage / COALESCE(conf_chargeable_unit_rate, conf_ecu_rate) * conf_storage_weight) + (querying / COALESCE(conf_chargeable_unit_rate, conf_ecu_rate) * conf_query_weight) + (indexing / COALESCE(conf_chargeable_unit_rate, conf_ecu_rate) * conf_indexing_weight)) / total_weight_hot, ((storage / COALESCE(conf_chargeable_unit_rate, conf_ecu_rate) * conf_storage_weight) + (querying / COALESCE(conf_chargeable_unit_rate, conf_ecu_rate) * conf_query_weight)) / total_weight_cold) * COALESCE(conf_chargeable_unit_rate, conf_ecu_rate)\n| STATS Indexing = SUM(indexing), Querying = SUM(querying), Storage = SUM(storage), Blended = SUM(blended) BY tier_label\n| WHERE Blended > 0\n| RENAME tier_label AS `Data Tier`\n| SORT Blended DESC" - }, - "visualization": { - "columns": [ - { - "columnId": "Data Tier", - "isMetric": false, - "isTransposed": false - }, - { - "columnId": "Indexing", - "isMetric": true, - "isTransposed": false, - "summaryRow": "sum" - }, - { - "columnId": "Querying", - "isMetric": true, - "isTransposed": false, - "summaryRow": "sum" - }, - { - "columnId": "Storage", - "isMetric": true, - "isTransposed": false, - "summaryRow": "sum" - }, - { - "columnId": "Blended", - "isMetric": true, - "isTransposed": false, - "summaryRow": "sum" - } - ], - "layerId": "9d199172-f9fa-4fc4-82af-c7e2672a3f54", - "layerType": "data" + "title": "Workload Mix by Data Tier", + "version": 2, + "visualizationType": "lnsDatatable" + }, + "description": "The allocatable pool redistributed across data tiers by workload \u2014 a weighted blend of indexing, query and storage share. The total is slightly below the realized pool because deployment-days that have a billing pool but no monitoring breakdown cannot be allocated to a tier.", + "drilldowns": [], + "title": "Workload Mix by Data Tier" + }, + "gridData": { + "h": 10, + "i": "b6c32121-c93d-4a4c-a850-7f486a476d81", + "sectionId": "d11ae80b-d7bb-474f-9082-81752ab2865d", + "w": 48, + "x": 0, + "y": 3 + }, + "panelIndex": "b6c32121-c93d-4a4c-a850-7f486a476d81", + "type": "lens" + }, + { + "embeddableConfig": { + "savedVis": { + "data": { + "aggs": [], + "searchSource": { + "filter": [], + "query": { + "language": "kuery", + "query": "" + } + } + }, + "description": "", + "params": { + "fontSize": 12, + "markdown": "## Indexing", + "openLinksInNewTab": false + }, + "title": "", + "type": "markdown", + "uiState": {} + }, + "title": "" + }, + "gridData": { + "h": 3, + "i": "41c6c963-16be-4b21-b59c-3c123f84ad57", + "sectionId": "d11ae80b-d7bb-474f-9082-81752ab2865d", + "w": 12, + "x": 0, + "y": 13 + }, + "panelIndex": "41c6c963-16be-4b21-b59c-3c123f84ad57", + "type": "visualization" + }, + { + "embeddableConfig": { + "savedVis": { + "data": { + "aggs": [], + "searchSource": { + "filter": [], + "query": { + "language": "kuery", + "query": "" + } + } + }, + "description": "", + "params": { + "fontSize": 12, + "markdown": "## Querying", + "openLinksInNewTab": false + }, + "title": "", + "type": "markdown", + "uiState": {} + }, + "title": "" + }, + "gridData": { + "h": 3, + "i": "1ad2d1fd-04c2-4d2b-ad5d-15ffb0176cc1", + "sectionId": "d11ae80b-d7bb-474f-9082-81752ab2865d", + "w": 12, + "x": 12, + "y": 13 + }, + "panelIndex": "1ad2d1fd-04c2-4d2b-ad5d-15ffb0176cc1", + "type": "visualization" + }, + { + "embeddableConfig": { + "savedVis": { + "data": { + "aggs": [], + "searchSource": { + "filter": [], + "query": { + "language": "kuery", + "query": "" + } + } + }, + "description": "", + "params": { + "fontSize": 12, + "markdown": "## Storage", + "openLinksInNewTab": false + }, + "title": "", + "type": "markdown", + "uiState": {} + }, + "title": "" + }, + "gridData": { + "h": 3, + "i": "2753c162-7420-4b8d-971f-4bda81d7be2b", + "sectionId": "d11ae80b-d7bb-474f-9082-81752ab2865d", + "w": 12, + "x": 24, + "y": 13 + }, + "panelIndex": "2753c162-7420-4b8d-971f-4bda81d7be2b", + "type": "visualization" + }, + { + "embeddableConfig": { + "savedVis": { + "data": { + "aggs": [], + "searchSource": { + "filter": [], + "query": { + "language": "kuery", + "query": "" + } + } + }, + "description": "", + "params": { + "fontSize": 12, + "markdown": "## Blended", + "openLinksInNewTab": false + }, + "title": "", + "type": "markdown", + "uiState": {} + }, + "title": "" + }, + "gridData": { + "h": 3, + "i": "c83bffe6-c501-430a-8769-a964b240acf5", + "sectionId": "d11ae80b-d7bb-474f-9082-81752ab2865d", + "w": 12, + "x": 36, + "y": 13 + }, + "panelIndex": "c83bffe6-c501-430a-8769-a964b240acf5", + "type": "visualization" + }, + { + "embeddableConfig": { + "attributes": { + "references": [], + "state": { + "datasourceStates": { + "textBased": { + "indexPatternRefs": [], + "layers": { + "0f6d5278-b325-49bc-9eb7-2febd2a5de0f": { + "columns": [ + { + "columnId": "agg_indexing", + "customLabel": true, + "fieldName": "agg_indexing", + "inMetricDimension": true, + "label": "Indexing", + "meta": { + "esType": "double", + "type": "number" + }, + "params": { + "format": { + "id": "number", + "params": { + "decimals": 2 + } } + } }, - "title": "Workload Mix by Data Tier", - "version": 2, - "visualizationType": "lnsDatatable" + { + "columnId": "datastream", + "customLabel": true, + "fieldName": "datastream", + "label": "Data stream", + "meta": { + "esType": "keyword", + "type": "string" + } + } + ], + "index": "chargeback_integration", + "query": { + "esql": "FROM billing_realized_pool_lookup\n| LOOKUP JOIN cluster_capacity_utilization_lookup ON composite_key\n| LOOKUP JOIN cluster_deployment_contribution_lookup ON composite_key\n| LOOKUP JOIN cluster_datastream_contribution_lookup ON composite_key\n| LOOKUP JOIN chargeback_conf_lookup ON @timestamp >= conf_start_date AND @timestamp <= conf_end_date\n| EVAL heap_util = COALESCE(heap_used_pct_p95, 100) / 100, disk_util = COALESCE(disk_used_pct_p95, 100) / 100, w_sum = conf_utilization_memory_weight + conf_utilization_storage_weight, capacity_utilization_score = CASE(w_sum > 0, (conf_utilization_memory_weight * heap_util + conf_utilization_storage_weight * disk_util) / w_sum, 1.0)\n| EVAL chargeable_pool = data_tier_capacity_ecu * COALESCE(capacity_utilization_score, 1.0), utilization_data_missing = heap_used_pct_p95 IS NULL AND disk_used_pct_p95 IS NULL\n| EVAL \n indexing = CASE (deployment_sum_indexing_time > 0, TO_DOUBLE(datastream_sum_indexing_time) / deployment_sum_indexing_time * chargeable_pool) * COALESCE(conf_chargeable_unit_rate, conf_ecu_rate)\n| STATS \n agg_indexing = sum(indexing)\n BY \n datastream\n| WHERE agg_indexing > 0\n| SORT agg_indexing DESC\n| Limit 20" + }, + "timeField": "@timestamp" + } + } + } + }, + "filters": [], + "needsRefresh": false, + "query": { + "esql": "FROM billing_realized_pool_lookup\n| LOOKUP JOIN cluster_capacity_utilization_lookup ON composite_key\n| LOOKUP JOIN cluster_deployment_contribution_lookup ON composite_key\n| LOOKUP JOIN cluster_datastream_contribution_lookup ON composite_key\n| LOOKUP JOIN chargeback_conf_lookup ON @timestamp >= conf_start_date AND @timestamp <= conf_end_date\n| EVAL heap_util = COALESCE(heap_used_pct_p95, 100) / 100, disk_util = COALESCE(disk_used_pct_p95, 100) / 100, w_sum = conf_utilization_memory_weight + conf_utilization_storage_weight, capacity_utilization_score = CASE(w_sum > 0, (conf_utilization_memory_weight * heap_util + conf_utilization_storage_weight * disk_util) / w_sum, 1.0)\n| EVAL chargeable_pool = data_tier_capacity_ecu * COALESCE(capacity_utilization_score, 1.0), utilization_data_missing = heap_used_pct_p95 IS NULL AND disk_used_pct_p95 IS NULL\n| EVAL \n indexing = CASE (deployment_sum_indexing_time > 0, TO_DOUBLE(datastream_sum_indexing_time) / deployment_sum_indexing_time * chargeable_pool) * COALESCE(conf_chargeable_unit_rate, conf_ecu_rate)\n| STATS \n agg_indexing = sum(indexing)\n BY \n datastream\n| WHERE agg_indexing > 0\n| SORT agg_indexing DESC\n| Limit 20" + }, + "visualization": { + "layers": [ + { + "categoryDisplay": "default", + "colorMapping": { + "assignments": [], + "colorMode": { + "type": "categorical" + }, + "paletteId": "eui_amsterdam_color_blind", + "specialAssignments": [ + { + "color": { + "type": "loop" + }, + "rules": [ + { + "type": "other" + } + ], + "touched": false + } + ] }, - "description": "The allocatable pool redistributed across data tiers by workload \u2014 a weighted blend of indexing, query and storage share. The total is slightly below the realized pool because deployment-days that have a billing pool but no monitoring breakdown cannot be allocated to a tier.", - "drilldowns": [], - "title": "Workload Mix by Data Tier" - }, - "gridData": { - "h": 10, - "i": "b6c32121-c93d-4a4c-a850-7f486a476d81", - "sectionId": "d11ae80b-d7bb-474f-9082-81752ab2865d", - "w": 48, - "x": 0, - "y": 3 - }, - "panelIndex": "b6c32121-c93d-4a4c-a850-7f486a476d81", - "type": "lens" + "emptySizeRatio": 0.3, + "layerId": "0f6d5278-b325-49bc-9eb7-2febd2a5de0f", + "layerType": "data", + "legendDisplay": "default", + "metrics": [ + "agg_indexing" + ], + "nestedLegend": false, + "numberDisplay": "percent", + "primaryGroups": [ + "datastream" + ] + } + ], + "shape": "donut" + } }, - { - "embeddableConfig": { - "savedVis": { - "data": { - "aggs": [], - "searchSource": { - "filter": [], - "query": { - "language": "kuery", - "query": "" - } + "title": "Top 20 Data Streams \u2014 Indexing Cost", + "version": 2, + "visualizationType": "lnsPie" + }, + "title": "Top 20 Data Streams \u2014 Indexing Cost" + }, + "gridData": { + "h": 15, + "i": "b349f8c6-846f-40d8-83e7-0b8365a7988e", + "sectionId": "d11ae80b-d7bb-474f-9082-81752ab2865d", + "w": 12, + "x": 0, + "y": 16 + }, + "panelIndex": "b349f8c6-846f-40d8-83e7-0b8365a7988e", + "type": "lens" + }, + { + "embeddableConfig": { + "attributes": { + "references": [], + "state": { + "datasourceStates": { + "textBased": { + "indexPatternRefs": [], + "layers": { + "3b03b6a8-35ae-4554-8bd6-a887d8b36748": { + "columns": [ + { + "columnId": "agg_querying", + "customLabel": true, + "fieldName": "agg_querying", + "inMetricDimension": true, + "label": "Querying", + "meta": { + "esType": "double", + "type": "number" + }, + "params": { + "format": { + "id": "number", + "params": { + "decimals": 2 + } } + } }, - "description": "", - "params": { - "fontSize": 12, - "markdown": "## Indexing", - "openLinksInNewTab": false - }, - "title": "", - "type": "markdown", - "uiState": {} + { + "columnId": "datastream", + "customLabel": true, + "fieldName": "datastream", + "label": "Data stream", + "meta": { + "esType": "keyword", + "type": "string" + } + } + ], + "index": "chargeback_integration", + "query": { + "esql": "FROM billing_realized_pool_lookup\n| LOOKUP JOIN cluster_capacity_utilization_lookup ON composite_key\n| LOOKUP JOIN cluster_deployment_contribution_lookup ON composite_key\n| LOOKUP JOIN cluster_datastream_contribution_lookup ON composite_key\n| LOOKUP JOIN chargeback_conf_lookup ON @timestamp >= conf_start_date AND @timestamp <= conf_end_date\n| EVAL heap_util = COALESCE(heap_used_pct_p95, 100) / 100, disk_util = COALESCE(disk_used_pct_p95, 100) / 100, w_sum = conf_utilization_memory_weight + conf_utilization_storage_weight, capacity_utilization_score = CASE(w_sum > 0, (conf_utilization_memory_weight * heap_util + conf_utilization_storage_weight * disk_util) / w_sum, 1.0)\n| EVAL chargeable_pool = data_tier_capacity_ecu * COALESCE(capacity_utilization_score, 1.0), utilization_data_missing = heap_used_pct_p95 IS NULL AND disk_used_pct_p95 IS NULL\n| EVAL \n querying = CASE (deployment_sum_query_time > 0, TO_DOUBLE(datastream_sum_query_time) / deployment_sum_query_time * chargeable_pool) * COALESCE(conf_chargeable_unit_rate, conf_ecu_rate)\n| STATS \n agg_querying = sum(querying)\n BY \n datastream\n| WHERE agg_querying > 0\n| SORT agg_querying DESC\n| Limit 20" + }, + "timeField": "@timestamp" + } + } + } + }, + "filters": [], + "needsRefresh": false, + "query": { + "esql": "FROM billing_realized_pool_lookup\n| LOOKUP JOIN cluster_capacity_utilization_lookup ON composite_key\n| LOOKUP JOIN cluster_deployment_contribution_lookup ON composite_key\n| LOOKUP JOIN cluster_datastream_contribution_lookup ON composite_key\n| LOOKUP JOIN chargeback_conf_lookup ON @timestamp >= conf_start_date AND @timestamp <= conf_end_date\n| EVAL heap_util = COALESCE(heap_used_pct_p95, 100) / 100, disk_util = COALESCE(disk_used_pct_p95, 100) / 100, w_sum = conf_utilization_memory_weight + conf_utilization_storage_weight, capacity_utilization_score = CASE(w_sum > 0, (conf_utilization_memory_weight * heap_util + conf_utilization_storage_weight * disk_util) / w_sum, 1.0)\n| EVAL chargeable_pool = data_tier_capacity_ecu * COALESCE(capacity_utilization_score, 1.0), utilization_data_missing = heap_used_pct_p95 IS NULL AND disk_used_pct_p95 IS NULL\n| EVAL \n querying = CASE (deployment_sum_query_time > 0, TO_DOUBLE(datastream_sum_query_time) / deployment_sum_query_time * chargeable_pool) * COALESCE(conf_chargeable_unit_rate, conf_ecu_rate)\n| STATS \n agg_querying = sum(querying)\n BY \n datastream\n| WHERE agg_querying > 0\n| SORT agg_querying DESC\n| Limit 20" + }, + "visualization": { + "layers": [ + { + "categoryDisplay": "default", + "colorMapping": { + "assignments": [], + "colorMode": { + "type": "categorical" + }, + "paletteId": "eui_amsterdam_color_blind", + "specialAssignments": [ + { + "color": { + "type": "loop" + }, + "rules": [ + { + "type": "other" + } + ], + "touched": false + } + ] }, - "title": "" - }, - "gridData": { - "h": 3, - "i": "41c6c963-16be-4b21-b59c-3c123f84ad57", - "sectionId": "d11ae80b-d7bb-474f-9082-81752ab2865d", - "w": 12, - "x": 0, - "y": 13 - }, - "panelIndex": "41c6c963-16be-4b21-b59c-3c123f84ad57", - "type": "visualization" + "emptySizeRatio": 0.3, + "layerId": "3b03b6a8-35ae-4554-8bd6-a887d8b36748", + "layerType": "data", + "legendDisplay": "default", + "metrics": [ + "agg_querying" + ], + "nestedLegend": false, + "numberDisplay": "percent", + "primaryGroups": [ + "datastream" + ] + } + ], + "shape": "donut" + } }, - { - "embeddableConfig": { - "savedVis": { - "data": { - "aggs": [], - "searchSource": { - "filter": [], - "query": { - "language": "kuery", - "query": "" - } + "title": "Top 20 Data Streams \u2014 Query Cost", + "version": 2, + "visualizationType": "lnsPie" + }, + "title": "Top 20 Data Streams \u2014 Query Cost" + }, + "gridData": { + "h": 15, + "i": "cd86b33a-2af1-4dbc-b296-f3e5a3ee5970", + "sectionId": "d11ae80b-d7bb-474f-9082-81752ab2865d", + "w": 12, + "x": 12, + "y": 16 + }, + "panelIndex": "cd86b33a-2af1-4dbc-b296-f3e5a3ee5970", + "type": "lens" + }, + { + "embeddableConfig": { + "attributes": { + "references": [], + "state": { + "datasourceStates": { + "textBased": { + "indexPatternRefs": [], + "layers": { + "2813628b-2a20-484c-9a19-11955d86e9ff": { + "columns": [ + { + "columnId": "agg_storage", + "customLabel": true, + "fieldName": "agg_storage", + "inMetricDimension": true, + "label": "Storage", + "meta": { + "esType": "double", + "type": "number" + }, + "params": { + "format": { + "id": "number", + "params": { + "decimals": 2 + } } + } }, - "description": "", - "params": { - "fontSize": 12, - "markdown": "## Querying", - "openLinksInNewTab": false - }, - "title": "", - "type": "markdown", - "uiState": {} + { + "columnId": "datastream", + "customLabel": true, + "fieldName": "datastream", + "label": "Data stream", + "meta": { + "esType": "keyword", + "type": "string" + } + } + ], + "index": "chargeback_integration", + "query": { + "esql": "FROM billing_realized_pool_lookup\n| LOOKUP JOIN cluster_capacity_utilization_lookup ON composite_key\n| LOOKUP JOIN cluster_deployment_contribution_lookup ON composite_key\n| LOOKUP JOIN cluster_datastream_contribution_lookup ON composite_key\n| LOOKUP JOIN chargeback_conf_lookup ON @timestamp >= conf_start_date AND @timestamp <= conf_end_date\n| EVAL heap_util = COALESCE(heap_used_pct_p95, 100) / 100, disk_util = COALESCE(disk_used_pct_p95, 100) / 100, w_sum = conf_utilization_memory_weight + conf_utilization_storage_weight, capacity_utilization_score = CASE(w_sum > 0, (conf_utilization_memory_weight * heap_util + conf_utilization_storage_weight * disk_util) / w_sum, 1.0)\n| EVAL chargeable_pool = data_tier_capacity_ecu * COALESCE(capacity_utilization_score, 1.0), utilization_data_missing = heap_used_pct_p95 IS NULL AND disk_used_pct_p95 IS NULL\n| EVAL \n data_set = CASE (deployment_sum_data_set_store_size > 0, TO_DOUBLE(datastream_sum_data_set_store_size) / deployment_sum_data_set_store_size * chargeable_pool),\n store = CASE (deployment_sum_store_size > 0, TO_DOUBLE(datastream_sum_store_size) / deployment_sum_store_size * chargeable_pool),\n storage = CASE (store == 0, data_set, store) * COALESCE(conf_chargeable_unit_rate, conf_ecu_rate)\n| STATS \n agg_storage = sum(storage)\n BY \n datastream\n| WHERE agg_storage > 0\n| SORT agg_storage DESC\n| limit 20" + }, + "timeField": "@timestamp" + } + } + } + }, + "filters": [], + "needsRefresh": false, + "query": { + "esql": "FROM billing_realized_pool_lookup\n| LOOKUP JOIN cluster_capacity_utilization_lookup ON composite_key\n| LOOKUP JOIN cluster_deployment_contribution_lookup ON composite_key\n| LOOKUP JOIN cluster_datastream_contribution_lookup ON composite_key\n| LOOKUP JOIN chargeback_conf_lookup ON @timestamp >= conf_start_date AND @timestamp <= conf_end_date\n| EVAL heap_util = COALESCE(heap_used_pct_p95, 100) / 100, disk_util = COALESCE(disk_used_pct_p95, 100) / 100, w_sum = conf_utilization_memory_weight + conf_utilization_storage_weight, capacity_utilization_score = CASE(w_sum > 0, (conf_utilization_memory_weight * heap_util + conf_utilization_storage_weight * disk_util) / w_sum, 1.0)\n| EVAL chargeable_pool = data_tier_capacity_ecu * COALESCE(capacity_utilization_score, 1.0), utilization_data_missing = heap_used_pct_p95 IS NULL AND disk_used_pct_p95 IS NULL\n| EVAL \n data_set = CASE (deployment_sum_data_set_store_size > 0, TO_DOUBLE(datastream_sum_data_set_store_size) / deployment_sum_data_set_store_size * chargeable_pool),\n store = CASE (deployment_sum_store_size > 0, TO_DOUBLE(datastream_sum_store_size) / deployment_sum_store_size * chargeable_pool),\n storage = CASE (store == 0, data_set, store) * COALESCE(conf_chargeable_unit_rate, conf_ecu_rate)\n| STATS \n agg_storage = sum(storage)\n BY \n datastream\n| WHERE agg_storage > 0\n| SORT agg_storage DESC\n| limit 20" + }, + "visualization": { + "layers": [ + { + "categoryDisplay": "default", + "colorMapping": { + "assignments": [], + "colorMode": { + "type": "categorical" + }, + "paletteId": "eui_amsterdam_color_blind", + "specialAssignments": [ + { + "color": { + "type": "loop" + }, + "rules": [ + { + "type": "other" + } + ], + "touched": false + } + ] }, - "title": "" - }, - "gridData": { - "h": 3, - "i": "1ad2d1fd-04c2-4d2b-ad5d-15ffb0176cc1", - "sectionId": "d11ae80b-d7bb-474f-9082-81752ab2865d", - "w": 12, - "x": 12, - "y": 13 - }, - "panelIndex": "1ad2d1fd-04c2-4d2b-ad5d-15ffb0176cc1", - "type": "visualization" + "emptySizeRatio": 0.3, + "layerId": "2813628b-2a20-484c-9a19-11955d86e9ff", + "layerType": "data", + "legendDisplay": "default", + "metrics": [ + "agg_storage" + ], + "nestedLegend": false, + "numberDisplay": "percent", + "primaryGroups": [ + "datastream" + ] + } + ], + "shape": "donut" + } }, - { - "embeddableConfig": { - "savedVis": { - "data": { - "aggs": [], - "searchSource": { - "filter": [], - "query": { - "language": "kuery", - "query": "" - } - } - }, - "description": "", - "params": { - "fontSize": 12, - "markdown": "## Storage", - "openLinksInNewTab": false + "title": "Top 20 Data Streams \u2014 Storage Cost", + "version": 2, + "visualizationType": "lnsPie" + }, + "title": "Top 20 Data Streams \u2014 Storage Cost" + }, + "gridData": { + "h": 15, + "i": "c3afc6bc-09c4-4f60-a038-163d535a7d32", + "sectionId": "d11ae80b-d7bb-474f-9082-81752ab2865d", + "w": 12, + "x": 24, + "y": 16 + }, + "panelIndex": "c3afc6bc-09c4-4f60-a038-163d535a7d32", + "type": "lens" + }, + { + "embeddableConfig": { + "attributes": { + "references": [], + "state": { + "datasourceStates": { + "textBased": { + "indexPatternRefs": [], + "layers": { + "ee58e17d-eba7-48a7-bae8-2df3607d9653": { + "columns": [ + { + "columnId": "agg_blended", + "customLabel": false, + "fieldName": "agg_blended", + "inMetricDimension": true, + "label": "agg_blended", + "meta": { + "esType": "double", + "type": "number" + } }, - "title": "", - "type": "markdown", - "uiState": {} + { + "columnId": "datastream", + "customLabel": false, + "fieldName": "datastream", + "label": "datastream", + "meta": { + "esType": "keyword", + "type": "string" + } + } + ], + "index": "chargeback_integration", + "query": { + "esql": "FROM billing_realized_pool_lookup\n| LOOKUP JOIN cluster_capacity_utilization_lookup ON composite_key\n| LOOKUP JOIN cluster_deployment_contribution_lookup ON composite_key\n| LOOKUP JOIN cluster_tier_and_datastream_contribution_lookup ON composite_key\n| LOOKUP JOIN chargeback_conf_lookup ON @timestamp >= conf_start_date AND @timestamp <= conf_end_date\n| EVAL heap_util = COALESCE(heap_used_pct_p95, 100) / 100, disk_util = COALESCE(disk_used_pct_p95, 100) / 100, w_sum = conf_utilization_memory_weight + conf_utilization_storage_weight, capacity_utilization_score = CASE(w_sum > 0, (conf_utilization_memory_weight * heap_util + conf_utilization_storage_weight * disk_util) / w_sum, 1.0)\n| EVAL chargeable_pool = data_tier_capacity_ecu * COALESCE(capacity_utilization_score, 1.0), utilization_data_missing = heap_used_pct_p95 IS NULL AND disk_used_pct_p95 IS NULL\n| EVAL\n indexing = CASE (deployment_sum_indexing_time > 0, TO_DOUBLE(tier_and_datastream_sum_indexing_time) / deployment_sum_indexing_time * chargeable_pool),\n querying = CASE (deployment_sum_query_time > 0, TO_DOUBLE(tier_and_datastream_sum_query_time) / deployment_sum_query_time * chargeable_pool),\n data_set = CASE (deployment_sum_data_set_store_size > 0, TO_DOUBLE(tier_and_datastream_sum_data_set_store_size) / deployment_sum_data_set_store_size * chargeable_pool),\n store = CASE (deployment_sum_store_size > 0, TO_DOUBLE(tier_and_datastream_sum_store_size) / deployment_sum_store_size * chargeable_pool),\n storage = CASE (store == 0, data_set, store),\n total_weight_hot = conf_storage_weight + conf_query_weight + conf_indexing_weight,\n total_weight_cold = conf_storage_weight + conf_query_weight,\n blended = CASE (\n tier == \"hot/content\",\n TO_DOUBLE(((storage * conf_storage_weight) + (querying * conf_query_weight) + (indexing * conf_indexing_weight))) / total_weight_hot,\n TO_DOUBLE(((storage * conf_storage_weight) + (querying * conf_query_weight))) / total_weight_cold\n ) * COALESCE(conf_chargeable_unit_rate, conf_ecu_rate)\n| STATS\n agg_blended = sum(blended)\n BY\n datastream\n| WHERE agg_blended > 0\n| SORT agg_blended DESC\n| limit 20" + }, + "timeField": "@timestamp" + } + } + } + }, + "filters": [], + "needsRefresh": false, + "query": { + "esql": "FROM billing_realized_pool_lookup\n| LOOKUP JOIN cluster_capacity_utilization_lookup ON composite_key\n| LOOKUP JOIN cluster_deployment_contribution_lookup ON composite_key\n| LOOKUP JOIN cluster_tier_and_datastream_contribution_lookup ON composite_key\n| LOOKUP JOIN chargeback_conf_lookup ON @timestamp >= conf_start_date AND @timestamp <= conf_end_date\n| EVAL heap_util = COALESCE(heap_used_pct_p95, 100) / 100, disk_util = COALESCE(disk_used_pct_p95, 100) / 100, w_sum = conf_utilization_memory_weight + conf_utilization_storage_weight, capacity_utilization_score = CASE(w_sum > 0, (conf_utilization_memory_weight * heap_util + conf_utilization_storage_weight * disk_util) / w_sum, 1.0)\n| EVAL chargeable_pool = data_tier_capacity_ecu * COALESCE(capacity_utilization_score, 1.0), utilization_data_missing = heap_used_pct_p95 IS NULL AND disk_used_pct_p95 IS NULL\n| EVAL\n indexing = CASE (deployment_sum_indexing_time > 0, TO_DOUBLE(tier_and_datastream_sum_indexing_time) / deployment_sum_indexing_time * chargeable_pool),\n querying = CASE (deployment_sum_query_time > 0, TO_DOUBLE(tier_and_datastream_sum_query_time) / deployment_sum_query_time * chargeable_pool),\n data_set = CASE (deployment_sum_data_set_store_size > 0, TO_DOUBLE(tier_and_datastream_sum_data_set_store_size) / deployment_sum_data_set_store_size * chargeable_pool),\n store = CASE (deployment_sum_store_size > 0, TO_DOUBLE(tier_and_datastream_sum_store_size) / deployment_sum_store_size * chargeable_pool),\n storage = CASE (store == 0, data_set, store),\n total_weight_hot = conf_storage_weight + conf_query_weight + conf_indexing_weight,\n total_weight_cold = conf_storage_weight + conf_query_weight,\n blended = CASE (\n tier == \"hot/content\",\n TO_DOUBLE(((storage * conf_storage_weight) + (querying * conf_query_weight) + (indexing * conf_indexing_weight))) / total_weight_hot,\n TO_DOUBLE(((storage * conf_storage_weight) + (querying * conf_query_weight))) / total_weight_cold\n ) * COALESCE(conf_chargeable_unit_rate, conf_ecu_rate)\n| STATS\n agg_blended = sum(blended)\n BY\n datastream\n| WHERE agg_blended > 0\n| SORT agg_blended DESC\n| limit 20" + }, + "visualization": { + "layers": [ + { + "categoryDisplay": "default", + "colorMapping": { + "assignments": [], + "colorMode": { + "type": "categorical" + }, + "paletteId": "default", + "specialAssignments": [ + { + "color": { + "type": "loop" + }, + "rules": [ + { + "type": "other" + } + ], + "touched": false + } + ] }, - "title": "" - }, - "gridData": { - "h": 3, - "i": "2753c162-7420-4b8d-971f-4bda81d7be2b", - "sectionId": "d11ae80b-d7bb-474f-9082-81752ab2865d", - "w": 12, - "x": 24, - "y": 13 - }, - "panelIndex": "2753c162-7420-4b8d-971f-4bda81d7be2b", - "type": "visualization" + "layerId": "ee58e17d-eba7-48a7-bae8-2df3607d9653", + "layerType": "data", + "legendDisplay": "default", + "metrics": [ + "agg_blended" + ], + "nestedLegend": false, + "numberDisplay": "percent", + "primaryGroups": [ + "datastream" + ] + } + ], + "shape": "pie" + } }, - { - "embeddableConfig": { - "savedVis": { - "data": { - "aggs": [], - "searchSource": { - "filter": [], - "query": { - "language": "kuery", - "query": "" - } - } - }, - "description": "", - "params": { - "fontSize": 12, - "markdown": "## Blended", - "openLinksInNewTab": false + "title": "Blended Cost by Data Stream", + "version": 2, + "visualizationType": "lnsPie" + }, + "title": "Blended Cost by Data Stream" + }, + "gridData": { + "h": 15, + "i": "e57765c6-40be-41cf-b3d2-e65ee8a51b67", + "sectionId": "d11ae80b-d7bb-474f-9082-81752ab2865d", + "w": 12, + "x": 36, + "y": 16 + }, + "panelIndex": "e57765c6-40be-41cf-b3d2-e65ee8a51b67", + "type": "lens" + }, + { + "embeddableConfig": { + "attributes": { + "references": [], + "state": { + "datasourceStates": { + "textBased": { + "indexPatternRefs": [], + "layers": { + "f183a66f-b6bc-4aff-9df9-a1ae032b17da": { + "columns": [ + { + "columnId": "agg_blended", + "customLabel": false, + "fieldName": "agg_blended", + "inMetricDimension": true, + "label": "agg_blended", + "meta": { + "esType": "double", + "type": "number" + } }, - "title": "", - "type": "markdown", - "uiState": {} + { + "columnId": "ds_namespace", + "customLabel": false, + "fieldName": "ds_namespace", + "label": "ds_namespace", + "meta": { + "esType": "keyword", + "type": "string" + } + } + ], + "index": "chargeback_integration", + "query": { + "esql": "FROM billing_realized_pool_lookup\n| LOOKUP JOIN cluster_capacity_utilization_lookup ON composite_key\n| LOOKUP JOIN cluster_deployment_contribution_lookup ON composite_key\n| LOOKUP JOIN cluster_tier_and_datastream_contribution_lookup ON composite_key\n| LOOKUP JOIN chargeback_conf_lookup ON @timestamp >= conf_start_date AND @timestamp <= conf_end_date\n| EVAL heap_util = COALESCE(heap_used_pct_p95, 100) / 100, disk_util = COALESCE(disk_used_pct_p95, 100) / 100, w_sum = conf_utilization_memory_weight + conf_utilization_storage_weight, capacity_utilization_score = CASE(w_sum > 0, (conf_utilization_memory_weight * heap_util + conf_utilization_storage_weight * disk_util) / w_sum, 1.0)\n| EVAL chargeable_pool = data_tier_capacity_ecu * COALESCE(capacity_utilization_score, 1.0), utilization_data_missing = heap_used_pct_p95 IS NULL AND disk_used_pct_p95 IS NULL\n| EVAL\n indexing = CASE (deployment_sum_indexing_time > 0, TO_DOUBLE(tier_and_datastream_sum_indexing_time) / deployment_sum_indexing_time * chargeable_pool),\n querying = CASE (deployment_sum_query_time > 0, TO_DOUBLE(tier_and_datastream_sum_query_time) / deployment_sum_query_time * chargeable_pool),\n data_set = CASE (deployment_sum_data_set_store_size > 0, TO_DOUBLE(tier_and_datastream_sum_data_set_store_size) / deployment_sum_data_set_store_size * chargeable_pool),\n store = CASE (deployment_sum_store_size > 0, TO_DOUBLE(tier_and_datastream_sum_store_size) / deployment_sum_store_size * chargeable_pool),\n storage = CASE (store == 0, data_set, store),\n total_weight_hot = conf_storage_weight + conf_query_weight + conf_indexing_weight,\n total_weight_cold = conf_storage_weight + conf_query_weight,\n blended = CASE (\n tier == \"hot/content\",\n TO_DOUBLE(((storage * conf_storage_weight) + (querying * conf_query_weight) + (indexing * conf_indexing_weight))) / total_weight_hot,\n TO_DOUBLE(((storage * conf_storage_weight) + (querying * conf_query_weight))) / total_weight_cold\n ) * COALESCE(conf_chargeable_unit_rate, conf_ecu_rate)\n| STATS\n agg_blended = sum(blended)\n BY\n ds_namespace\n| WHERE agg_blended > 0\n| SORT agg_blended DESC\n| limit 20" + }, + "timeField": "@timestamp" + } + } + } + }, + "filters": [], + "needsRefresh": false, + "query": { + "esql": "FROM billing_realized_pool_lookup\n| LOOKUP JOIN cluster_capacity_utilization_lookup ON composite_key\n| LOOKUP JOIN cluster_deployment_contribution_lookup ON composite_key\n| LOOKUP JOIN cluster_tier_and_datastream_contribution_lookup ON composite_key\n| LOOKUP JOIN chargeback_conf_lookup ON @timestamp >= conf_start_date AND @timestamp <= conf_end_date\n| EVAL heap_util = COALESCE(heap_used_pct_p95, 100) / 100, disk_util = COALESCE(disk_used_pct_p95, 100) / 100, w_sum = conf_utilization_memory_weight + conf_utilization_storage_weight, capacity_utilization_score = CASE(w_sum > 0, (conf_utilization_memory_weight * heap_util + conf_utilization_storage_weight * disk_util) / w_sum, 1.0)\n| EVAL chargeable_pool = data_tier_capacity_ecu * COALESCE(capacity_utilization_score, 1.0), utilization_data_missing = heap_used_pct_p95 IS NULL AND disk_used_pct_p95 IS NULL\n| EVAL\n indexing = CASE (deployment_sum_indexing_time > 0, TO_DOUBLE(tier_and_datastream_sum_indexing_time) / deployment_sum_indexing_time * chargeable_pool),\n querying = CASE (deployment_sum_query_time > 0, TO_DOUBLE(tier_and_datastream_sum_query_time) / deployment_sum_query_time * chargeable_pool),\n data_set = CASE (deployment_sum_data_set_store_size > 0, TO_DOUBLE(tier_and_datastream_sum_data_set_store_size) / deployment_sum_data_set_store_size * chargeable_pool),\n store = CASE (deployment_sum_store_size > 0, TO_DOUBLE(tier_and_datastream_sum_store_size) / deployment_sum_store_size * chargeable_pool),\n storage = CASE (store == 0, data_set, store),\n total_weight_hot = conf_storage_weight + conf_query_weight + conf_indexing_weight,\n total_weight_cold = conf_storage_weight + conf_query_weight,\n blended = CASE (\n tier == \"hot/content\",\n TO_DOUBLE(((storage * conf_storage_weight) + (querying * conf_query_weight) + (indexing * conf_indexing_weight))) / total_weight_hot,\n TO_DOUBLE(((storage * conf_storage_weight) + (querying * conf_query_weight))) / total_weight_cold\n ) * COALESCE(conf_chargeable_unit_rate, conf_ecu_rate)\n| STATS\n agg_blended = sum(blended)\n BY\n ds_namespace\n| WHERE agg_blended > 0\n| SORT agg_blended DESC\n| limit 20" + }, + "visualization": { + "layers": [ + { + "categoryDisplay": "default", + "colorMapping": { + "assignments": [], + "colorMode": { + "type": "categorical" + }, + "paletteId": "default", + "specialAssignments": [ + { + "color": { + "type": "loop" + }, + "rules": [ + { + "type": "other" + } + ], + "touched": false + } + ] }, - "title": "" - }, - "gridData": { - "h": 3, - "i": "c83bffe6-c501-430a-8769-a964b240acf5", - "sectionId": "d11ae80b-d7bb-474f-9082-81752ab2865d", - "w": 12, - "x": 36, - "y": 13 - }, - "panelIndex": "c83bffe6-c501-430a-8769-a964b240acf5", - "type": "visualization" + "layerId": "f183a66f-b6bc-4aff-9df9-a1ae032b17da", + "layerType": "data", + "legendDisplay": "default", + "metrics": [ + "agg_blended" + ], + "nestedLegend": false, + "numberDisplay": "percent", + "primaryGroups": [ + "ds_namespace" + ] + } + ], + "shape": "pie" + } }, - { - "embeddableConfig": { - "attributes": { - "references": [], - "state": { - "datasourceStates": { - "textBased": { - "indexPatternRefs": [], - "layers": { - "0f6d5278-b325-49bc-9eb7-2febd2a5de0f": { - "columns": [ - { - "columnId": "agg_indexing", - "customLabel": true, - "fieldName": "agg_indexing", - "inMetricDimension": true, - "label": "Indexing", - "meta": { - "esType": "double", - "type": "number" - }, - "params": { - "format": { - "id": "number", - "params": { - "decimals": 2 - } - } - } - }, - { - "columnId": "datastream", - "customLabel": true, - "fieldName": "datastream", - "label": "Data stream", - "meta": { - "esType": "keyword", - "type": "string" - } - } - ], - "index": "chargeback_integration", - "query": { - "esql": "FROM billing_realized_pool_lookup\n| LOOKUP JOIN cluster_capacity_utilization_lookup ON composite_key\n| LOOKUP JOIN cluster_deployment_contribution_lookup ON composite_key\n| LOOKUP JOIN cluster_datastream_contribution_lookup ON composite_key\n| LOOKUP JOIN chargeback_conf_lookup ON @timestamp >= conf_start_date AND @timestamp <= conf_end_date\n| EVAL heap_util = COALESCE(heap_used_pct_p95, 100) / 100, disk_util = COALESCE(disk_used_pct_p95, 100) / 100, w_sum = conf_utilization_memory_weight + conf_utilization_storage_weight, capacity_utilization_score = CASE(w_sum > 0, (conf_utilization_memory_weight * heap_util + conf_utilization_storage_weight * disk_util) / w_sum, 1.0)\n| EVAL chargeable_pool = data_tier_capacity_ecu * COALESCE(capacity_utilization_score, 1.0), utilization_data_missing = heap_used_pct_p95 IS NULL AND disk_used_pct_p95 IS NULL\n| EVAL \n indexing = CASE (deployment_sum_indexing_time > 0, TO_DOUBLE(datastream_sum_indexing_time) / deployment_sum_indexing_time * chargeable_pool) * COALESCE(conf_chargeable_unit_rate, conf_ecu_rate)\n| STATS \n agg_indexing = sum(indexing)\n BY \n datastream\n| WHERE agg_indexing > 0\n| SORT agg_indexing DESC\n| Limit 20" - }, - "timeField": "@timestamp" - } - } - } - }, - "filters": [], - "needsRefresh": false, - "query": { - "esql": "FROM billing_realized_pool_lookup\n| LOOKUP JOIN cluster_capacity_utilization_lookup ON composite_key\n| LOOKUP JOIN cluster_deployment_contribution_lookup ON composite_key\n| LOOKUP JOIN cluster_datastream_contribution_lookup ON composite_key\n| LOOKUP JOIN chargeback_conf_lookup ON @timestamp >= conf_start_date AND @timestamp <= conf_end_date\n| EVAL heap_util = COALESCE(heap_used_pct_p95, 100) / 100, disk_util = COALESCE(disk_used_pct_p95, 100) / 100, w_sum = conf_utilization_memory_weight + conf_utilization_storage_weight, capacity_utilization_score = CASE(w_sum > 0, (conf_utilization_memory_weight * heap_util + conf_utilization_storage_weight * disk_util) / w_sum, 1.0)\n| EVAL chargeable_pool = data_tier_capacity_ecu * COALESCE(capacity_utilization_score, 1.0), utilization_data_missing = heap_used_pct_p95 IS NULL AND disk_used_pct_p95 IS NULL\n| EVAL \n indexing = CASE (deployment_sum_indexing_time > 0, TO_DOUBLE(datastream_sum_indexing_time) / deployment_sum_indexing_time * chargeable_pool) * COALESCE(conf_chargeable_unit_rate, conf_ecu_rate)\n| STATS \n agg_indexing = sum(indexing)\n BY \n datastream\n| WHERE agg_indexing > 0\n| SORT agg_indexing DESC\n| Limit 20" - }, - "visualization": { - "layers": [ - { - "categoryDisplay": "default", - "colorMapping": { - "assignments": [], - "colorMode": { - "type": "categorical" - }, - "paletteId": "eui_amsterdam_color_blind", - "specialAssignments": [ - { - "color": { - "type": "loop" - }, - "rules": [ - { - "type": "other" - } - ], - "touched": false - } - ] - }, - "emptySizeRatio": 0.3, - "layerId": "0f6d5278-b325-49bc-9eb7-2febd2a5de0f", - "layerType": "data", - "legendDisplay": "default", - "metrics": [ - "agg_indexing" - ], - "nestedLegend": false, - "numberDisplay": "percent", - "primaryGroups": [ - "datastream" - ] - } - ], - "shape": "donut" - } + "title": "Blended Cost by Data Stream Namespace", + "version": 2, + "visualizationType": "lnsPie" + }, + "title": "Blended Cost by Data Stream Namespace" + }, + "gridData": { + "i": "9f4b6c49-4810-417c-978a-bb8d53a00497", + "sectionId": "d11ae80b-d7bb-474f-9082-81752ab2865d", + "x": 0, + "y": 31, + "w": 24, + "h": 15 + }, + "panelIndex": "0f7940f8-4b66-48b8-882d-6e85e48164db", + "type": "lens" + }, + { + "embeddableConfig": { + "attributes": { + "references": [], + "state": { + "datasourceStates": { + "textBased": { + "indexPatternRefs": [], + "layers": { + "3e7781df-333a-4e73-9c34-4472da1db16a": { + "columns": [ + { + "columnId": "agg_blended", + "customLabel": false, + "fieldName": "agg_blended", + "inMetricDimension": true, + "label": "agg_blended", + "meta": { + "esType": "double", + "type": "number" + } }, - "title": "Top 20 Data Streams \u2014 Indexing Cost", - "version": 2, - "visualizationType": "lnsPie" + { + "columnId": "ds_type", + "customLabel": false, + "fieldName": "ds_type", + "label": "ds_type", + "meta": { + "esType": "keyword", + "type": "string" + } + } + ], + "index": "chargeback_integration", + "query": { + "esql": "FROM billing_realized_pool_lookup\n| LOOKUP JOIN cluster_capacity_utilization_lookup ON composite_key\n| LOOKUP JOIN cluster_deployment_contribution_lookup ON composite_key\n| LOOKUP JOIN cluster_tier_and_datastream_contribution_lookup ON composite_key\n| LOOKUP JOIN chargeback_conf_lookup ON @timestamp >= conf_start_date AND @timestamp <= conf_end_date\n| EVAL heap_util = COALESCE(heap_used_pct_p95, 100) / 100, disk_util = COALESCE(disk_used_pct_p95, 100) / 100, w_sum = conf_utilization_memory_weight + conf_utilization_storage_weight, capacity_utilization_score = CASE(w_sum > 0, (conf_utilization_memory_weight * heap_util + conf_utilization_storage_weight * disk_util) / w_sum, 1.0)\n| EVAL chargeable_pool = data_tier_capacity_ecu * COALESCE(capacity_utilization_score, 1.0), utilization_data_missing = heap_used_pct_p95 IS NULL AND disk_used_pct_p95 IS NULL\n| EVAL\n indexing = CASE (deployment_sum_indexing_time > 0, TO_DOUBLE(tier_and_datastream_sum_indexing_time) / deployment_sum_indexing_time * chargeable_pool),\n querying = CASE (deployment_sum_query_time > 0, TO_DOUBLE(tier_and_datastream_sum_query_time) / deployment_sum_query_time * chargeable_pool),\n data_set = CASE (deployment_sum_data_set_store_size > 0, TO_DOUBLE(tier_and_datastream_sum_data_set_store_size) / deployment_sum_data_set_store_size * chargeable_pool),\n store = CASE (deployment_sum_store_size > 0, TO_DOUBLE(tier_and_datastream_sum_store_size) / deployment_sum_store_size * chargeable_pool),\n storage = CASE (store == 0, data_set, store),\n total_weight_hot = conf_storage_weight + conf_query_weight + conf_indexing_weight,\n total_weight_cold = conf_storage_weight + conf_query_weight,\n blended = CASE (\n tier == \"hot/content\",\n TO_DOUBLE(((storage * conf_storage_weight) + (querying * conf_query_weight) + (indexing * conf_indexing_weight))) / total_weight_hot,\n TO_DOUBLE(((storage * conf_storage_weight) + (querying * conf_query_weight))) / total_weight_cold\n ) * COALESCE(conf_chargeable_unit_rate, conf_ecu_rate)\n| STATS\n agg_blended = sum(blended)\n BY\n ds_type\n| WHERE agg_blended > 0\n| SORT agg_blended DESC\n| limit 20" + }, + "timeField": "@timestamp" + } + } + } + }, + "filters": [], + "needsRefresh": false, + "query": { + "esql": "FROM billing_realized_pool_lookup\n| LOOKUP JOIN cluster_capacity_utilization_lookup ON composite_key\n| LOOKUP JOIN cluster_deployment_contribution_lookup ON composite_key\n| LOOKUP JOIN cluster_tier_and_datastream_contribution_lookup ON composite_key\n| LOOKUP JOIN chargeback_conf_lookup ON @timestamp >= conf_start_date AND @timestamp <= conf_end_date\n| EVAL heap_util = COALESCE(heap_used_pct_p95, 100) / 100, disk_util = COALESCE(disk_used_pct_p95, 100) / 100, w_sum = conf_utilization_memory_weight + conf_utilization_storage_weight, capacity_utilization_score = CASE(w_sum > 0, (conf_utilization_memory_weight * heap_util + conf_utilization_storage_weight * disk_util) / w_sum, 1.0)\n| EVAL chargeable_pool = data_tier_capacity_ecu * COALESCE(capacity_utilization_score, 1.0), utilization_data_missing = heap_used_pct_p95 IS NULL AND disk_used_pct_p95 IS NULL\n| EVAL\n indexing = CASE (deployment_sum_indexing_time > 0, TO_DOUBLE(tier_and_datastream_sum_indexing_time) / deployment_sum_indexing_time * chargeable_pool),\n querying = CASE (deployment_sum_query_time > 0, TO_DOUBLE(tier_and_datastream_sum_query_time) / deployment_sum_query_time * chargeable_pool),\n data_set = CASE (deployment_sum_data_set_store_size > 0, TO_DOUBLE(tier_and_datastream_sum_data_set_store_size) / deployment_sum_data_set_store_size * chargeable_pool),\n store = CASE (deployment_sum_store_size > 0, TO_DOUBLE(tier_and_datastream_sum_store_size) / deployment_sum_store_size * chargeable_pool),\n storage = CASE (store == 0, data_set, store),\n total_weight_hot = conf_storage_weight + conf_query_weight + conf_indexing_weight,\n total_weight_cold = conf_storage_weight + conf_query_weight,\n blended = CASE (\n tier == \"hot/content\",\n TO_DOUBLE(((storage * conf_storage_weight) + (querying * conf_query_weight) + (indexing * conf_indexing_weight))) / total_weight_hot,\n TO_DOUBLE(((storage * conf_storage_weight) + (querying * conf_query_weight))) / total_weight_cold\n ) * COALESCE(conf_chargeable_unit_rate, conf_ecu_rate)\n| STATS\n agg_blended = sum(blended)\n BY\n ds_type\n| WHERE agg_blended > 0\n| SORT agg_blended DESC\n| limit 20" + }, + "visualization": { + "layers": [ + { + "categoryDisplay": "default", + "colorMapping": { + "assignments": [], + "colorMode": { + "type": "categorical" + }, + "paletteId": "default", + "specialAssignments": [ + { + "color": { + "type": "loop" + }, + "rules": [ + { + "type": "other" + } + ], + "touched": false + } + ] }, - "title": "Top 20 Data Streams \u2014 Indexing Cost" - }, - "gridData": { - "h": 15, - "i": "b349f8c6-846f-40d8-83e7-0b8365a7988e", - "sectionId": "d11ae80b-d7bb-474f-9082-81752ab2865d", - "w": 12, - "x": 0, - "y": 16 - }, - "panelIndex": "b349f8c6-846f-40d8-83e7-0b8365a7988e", - "type": "lens" + "layerId": "3e7781df-333a-4e73-9c34-4472da1db16a", + "layerType": "data", + "legendDisplay": "default", + "metrics": [ + "agg_blended" + ], + "nestedLegend": false, + "numberDisplay": "percent", + "primaryGroups": [ + "ds_type" + ] + } + ], + "shape": "pie" + } }, - { - "embeddableConfig": { - "attributes": { - "references": [], - "state": { - "datasourceStates": { - "textBased": { - "indexPatternRefs": [], - "layers": { - "3b03b6a8-35ae-4554-8bd6-a887d8b36748": { - "columns": [ - { - "columnId": "agg_querying", - "customLabel": true, - "fieldName": "agg_querying", - "inMetricDimension": true, - "label": "Querying", - "meta": { - "esType": "double", - "type": "number" - }, - "params": { - "format": { - "id": "number", - "params": { - "decimals": 2 - } - } - } - }, - { - "columnId": "datastream", - "customLabel": true, - "fieldName": "datastream", - "label": "Data stream", - "meta": { - "esType": "keyword", - "type": "string" - } - } - ], - "index": "chargeback_integration", - "query": { - "esql": "FROM billing_realized_pool_lookup\n| LOOKUP JOIN cluster_capacity_utilization_lookup ON composite_key\n| LOOKUP JOIN cluster_deployment_contribution_lookup ON composite_key\n| LOOKUP JOIN cluster_datastream_contribution_lookup ON composite_key\n| LOOKUP JOIN chargeback_conf_lookup ON @timestamp >= conf_start_date AND @timestamp <= conf_end_date\n| EVAL heap_util = COALESCE(heap_used_pct_p95, 100) / 100, disk_util = COALESCE(disk_used_pct_p95, 100) / 100, w_sum = conf_utilization_memory_weight + conf_utilization_storage_weight, capacity_utilization_score = CASE(w_sum > 0, (conf_utilization_memory_weight * heap_util + conf_utilization_storage_weight * disk_util) / w_sum, 1.0)\n| EVAL chargeable_pool = data_tier_capacity_ecu * COALESCE(capacity_utilization_score, 1.0), utilization_data_missing = heap_used_pct_p95 IS NULL AND disk_used_pct_p95 IS NULL\n| EVAL \n querying = CASE (deployment_sum_query_time > 0, TO_DOUBLE(datastream_sum_query_time) / deployment_sum_query_time * chargeable_pool) * COALESCE(conf_chargeable_unit_rate, conf_ecu_rate)\n| STATS \n agg_querying = sum(querying)\n BY \n datastream\n| WHERE agg_querying > 0\n| SORT agg_querying DESC\n| Limit 20" - }, - "timeField": "@timestamp" - } - } - } + "title": "Blended Cost by Data Stream Type", + "version": 2, + "visualizationType": "lnsPie" + }, + "title": "Blended Cost by Data Stream Type" + }, + "gridData": { + "i": "e12e09a0-c446-4877-8868-b70883f82790", + "sectionId": "d11ae80b-d7bb-474f-9082-81752ab2865d", + "x": 24, + "y": 31, + "w": 24, + "h": 15 + }, + "panelIndex": "8b4899dc-c64f-4f25-af97-d817639ec4c2", + "type": "lens" + }, + { + "embeddableConfig": { + "attributes": { + "references": [], + "state": { + "datasourceStates": { + "textBased": { + "indexPatternRefs": [], + "layers": { + "23693fb0-43a5-4f95-bd77-d6a54e18e34a": { + "allColumns": [ + { + "columnId": "Data tier", + "customLabel": true, + "fieldName": "Data tier", + "inMetricDimension": true, + "label": "Data Tier", + "meta": { + "esType": "keyword", + "type": "string" + } + }, + { + "columnId": "Indexing", + "customLabel": false, + "fieldName": "Indexing", + "inMetricDimension": true, + "label": "Indexing", + "meta": { + "esType": "double", + "type": "number" + } + }, + { + "columnId": "Querying", + "customLabel": false, + "fieldName": "Querying", + "inMetricDimension": true, + "label": "Querying", + "meta": { + "esType": "double", + "type": "number" + } + }, + { + "columnId": "Storage", + "customLabel": false, + "fieldName": "Storage", + "inMetricDimension": true, + "label": "Storage", + "meta": { + "esType": "double", + "type": "number" + } + }, + { + "columnId": "ce5b4b7d-63cc-4961-9d04-657a81926514", + "fieldName": "Data stream", + "meta": { + "esType": "keyword", + "type": "string" + } + }, + { + "columnId": "Data stream", + "fieldName": "Data stream", + "label": "Data stream", + "meta": { + "esType": "keyword", + "type": "string" + } + } + ], + "columns": [ + { + "columnId": "ce5b4b7d-63cc-4961-9d04-657a81926514", + "fieldName": "Data stream", + "meta": { + "esType": "keyword", + "type": "string" + } + }, + { + "columnId": "3433219b-e0b9-4ee9-bf82-e2d00fc614ed", + "fieldName": "Data tier", + "meta": { + "esType": "keyword", + "type": "string" + } + }, + { + "columnId": "f649e7a2-4d2c-45a3-aa64-99f2e29c2dd0", + "fieldName": "Indexing", + "label": "Indexing", + "meta": { + "esType": "double", + "params": { + "id": "number" }, - "filters": [], - "needsRefresh": false, - "query": { - "esql": "FROM billing_realized_pool_lookup\n| LOOKUP JOIN cluster_capacity_utilization_lookup ON composite_key\n| LOOKUP JOIN cluster_deployment_contribution_lookup ON composite_key\n| LOOKUP JOIN cluster_datastream_contribution_lookup ON composite_key\n| LOOKUP JOIN chargeback_conf_lookup ON @timestamp >= conf_start_date AND @timestamp <= conf_end_date\n| EVAL heap_util = COALESCE(heap_used_pct_p95, 100) / 100, disk_util = COALESCE(disk_used_pct_p95, 100) / 100, w_sum = conf_utilization_memory_weight + conf_utilization_storage_weight, capacity_utilization_score = CASE(w_sum > 0, (conf_utilization_memory_weight * heap_util + conf_utilization_storage_weight * disk_util) / w_sum, 1.0)\n| EVAL chargeable_pool = data_tier_capacity_ecu * COALESCE(capacity_utilization_score, 1.0), utilization_data_missing = heap_used_pct_p95 IS NULL AND disk_used_pct_p95 IS NULL\n| EVAL \n querying = CASE (deployment_sum_query_time > 0, TO_DOUBLE(datastream_sum_query_time) / deployment_sum_query_time * chargeable_pool) * COALESCE(conf_chargeable_unit_rate, conf_ecu_rate)\n| STATS \n agg_querying = sum(querying)\n BY \n datastream\n| WHERE agg_querying > 0\n| SORT agg_querying DESC\n| Limit 20" + "sourceParams": { + "indexPattern": "billing_realized_pool_lookup", + "sourceField": "Indexing" }, - "visualization": { - "layers": [ - { - "categoryDisplay": "default", - "colorMapping": { - "assignments": [], - "colorMode": { - "type": "categorical" - }, - "paletteId": "eui_amsterdam_color_blind", - "specialAssignments": [ - { - "color": { - "type": "loop" - }, - "rules": [ - { - "type": "other" - } - ], - "touched": false - } - ] - }, - "emptySizeRatio": 0.3, - "layerId": "3b03b6a8-35ae-4554-8bd6-a887d8b36748", - "layerType": "data", - "legendDisplay": "default", - "metrics": [ - "agg_querying" - ], - "nestedLegend": false, - "numberDisplay": "percent", - "primaryGroups": [ - "datastream" - ] - } - ], - "shape": "donut" - } + "type": "number" + } }, - "title": "Top 20 Data Streams \u2014 Query Cost", - "version": 2, - "visualizationType": "lnsPie" - }, - "title": "Top 20 Data Streams \u2014 Query Cost" - }, - "gridData": { - "h": 15, - "i": "cd86b33a-2af1-4dbc-b296-f3e5a3ee5970", - "sectionId": "d11ae80b-d7bb-474f-9082-81752ab2865d", - "w": 12, - "x": 12, - "y": 16 - }, - "panelIndex": "cd86b33a-2af1-4dbc-b296-f3e5a3ee5970", - "type": "lens" - }, - { - "embeddableConfig": { - "attributes": { - "references": [], - "state": { - "datasourceStates": { - "textBased": { - "indexPatternRefs": [], - "layers": { - "2813628b-2a20-484c-9a19-11955d86e9ff": { - "columns": [ - { - "columnId": "agg_storage", - "customLabel": true, - "fieldName": "agg_storage", - "inMetricDimension": true, - "label": "Storage", - "meta": { - "esType": "double", - "type": "number" - }, - "params": { - "format": { - "id": "number", - "params": { - "decimals": 2 - } - } - } - }, - { - "columnId": "datastream", - "customLabel": true, - "fieldName": "datastream", - "label": "Data stream", - "meta": { - "esType": "keyword", - "type": "string" - } - } - ], - "index": "chargeback_integration", - "query": { - "esql": "FROM billing_realized_pool_lookup\n| LOOKUP JOIN cluster_capacity_utilization_lookup ON composite_key\n| LOOKUP JOIN cluster_deployment_contribution_lookup ON composite_key\n| LOOKUP JOIN cluster_datastream_contribution_lookup ON composite_key\n| LOOKUP JOIN chargeback_conf_lookup ON @timestamp >= conf_start_date AND @timestamp <= conf_end_date\n| EVAL heap_util = COALESCE(heap_used_pct_p95, 100) / 100, disk_util = COALESCE(disk_used_pct_p95, 100) / 100, w_sum = conf_utilization_memory_weight + conf_utilization_storage_weight, capacity_utilization_score = CASE(w_sum > 0, (conf_utilization_memory_weight * heap_util + conf_utilization_storage_weight * disk_util) / w_sum, 1.0)\n| EVAL chargeable_pool = data_tier_capacity_ecu * COALESCE(capacity_utilization_score, 1.0), utilization_data_missing = heap_used_pct_p95 IS NULL AND disk_used_pct_p95 IS NULL\n| EVAL \n data_set = CASE (deployment_sum_data_set_store_size > 0, TO_DOUBLE(datastream_sum_data_set_store_size) / deployment_sum_data_set_store_size * chargeable_pool),\n store = CASE (deployment_sum_store_size > 0, TO_DOUBLE(datastream_sum_store_size) / deployment_sum_store_size * chargeable_pool),\n storage = CASE (store == 0, data_set, store) * COALESCE(conf_chargeable_unit_rate, conf_ecu_rate)\n| STATS \n agg_storage = sum(storage)\n BY \n datastream\n| WHERE agg_storage > 0\n| SORT agg_storage DESC\n| limit 20" - }, - "timeField": "@timestamp" - } - } - } + { + "columnId": "71737902-035d-4164-978e-2614106901ea", + "fieldName": "Querying", + "label": "Querying", + "meta": { + "esType": "double", + "params": { + "id": "number" }, - "filters": [], - "needsRefresh": false, - "query": { - "esql": "FROM billing_realized_pool_lookup\n| LOOKUP JOIN cluster_capacity_utilization_lookup ON composite_key\n| LOOKUP JOIN cluster_deployment_contribution_lookup ON composite_key\n| LOOKUP JOIN cluster_datastream_contribution_lookup ON composite_key\n| LOOKUP JOIN chargeback_conf_lookup ON @timestamp >= conf_start_date AND @timestamp <= conf_end_date\n| EVAL heap_util = COALESCE(heap_used_pct_p95, 100) / 100, disk_util = COALESCE(disk_used_pct_p95, 100) / 100, w_sum = conf_utilization_memory_weight + conf_utilization_storage_weight, capacity_utilization_score = CASE(w_sum > 0, (conf_utilization_memory_weight * heap_util + conf_utilization_storage_weight * disk_util) / w_sum, 1.0)\n| EVAL chargeable_pool = data_tier_capacity_ecu * COALESCE(capacity_utilization_score, 1.0), utilization_data_missing = heap_used_pct_p95 IS NULL AND disk_used_pct_p95 IS NULL\n| EVAL \n data_set = CASE (deployment_sum_data_set_store_size > 0, TO_DOUBLE(datastream_sum_data_set_store_size) / deployment_sum_data_set_store_size * chargeable_pool),\n store = CASE (deployment_sum_store_size > 0, TO_DOUBLE(datastream_sum_store_size) / deployment_sum_store_size * chargeable_pool),\n storage = CASE (store == 0, data_set, store) * COALESCE(conf_chargeable_unit_rate, conf_ecu_rate)\n| STATS \n agg_storage = sum(storage)\n BY \n datastream\n| WHERE agg_storage > 0\n| SORT agg_storage DESC\n| limit 20" + "sourceParams": { + "indexPattern": "billing_realized_pool_lookup", + "sourceField": "Querying" }, - "visualization": { - "layers": [ - { - "categoryDisplay": "default", - "colorMapping": { - "assignments": [], - "colorMode": { - "type": "categorical" - }, - "paletteId": "eui_amsterdam_color_blind", - "specialAssignments": [ - { - "color": { - "type": "loop" - }, - "rules": [ - { - "type": "other" - } - ], - "touched": false - } - ] - }, - "emptySizeRatio": 0.3, - "layerId": "2813628b-2a20-484c-9a19-11955d86e9ff", - "layerType": "data", - "legendDisplay": "default", - "metrics": [ - "agg_storage" - ], - "nestedLegend": false, - "numberDisplay": "percent", - "primaryGroups": [ - "datastream" - ] - } - ], - "shape": "donut" - } + "type": "number" + } }, - "title": "Top 20 Data Streams \u2014 Storage Cost", - "version": 2, - "visualizationType": "lnsPie" - }, - "title": "Top 20 Data Streams \u2014 Storage Cost" - }, - "gridData": { - "h": 15, - "i": "c3afc6bc-09c4-4f60-a038-163d535a7d32", - "sectionId": "d11ae80b-d7bb-474f-9082-81752ab2865d", - "w": 12, - "x": 24, - "y": 16 - }, - "panelIndex": "c3afc6bc-09c4-4f60-a038-163d535a7d32", - "type": "lens" - }, - { - "embeddableConfig": { - "attributes": { - "references": [], - "state": { - "datasourceStates": { - "textBased": { - "indexPatternRefs": [], - "layers": { - "ee58e17d-eba7-48a7-bae8-2df3607d9653": { - "columns": [ - { - "columnId": "agg_blended", - "customLabel": false, - "fieldName": "agg_blended", - "inMetricDimension": true, - "label": "agg_blended", - "meta": { - "esType": "double", - "type": "number" - } - }, - { - "columnId": "datastream", - "customLabel": false, - "fieldName": "datastream", - "label": "datastream", - "meta": { - "esType": "keyword", - "type": "string" - } - } - ], - "index": "chargeback_integration", - "query": { - "esql": "FROM billing_realized_pool_lookup\n| LOOKUP JOIN cluster_capacity_utilization_lookup ON composite_key\n| LOOKUP JOIN cluster_deployment_contribution_lookup ON composite_key\n| LOOKUP JOIN cluster_tier_and_datastream_contribution_lookup ON composite_key\n| LOOKUP JOIN chargeback_conf_lookup ON @timestamp >= conf_start_date AND @timestamp <= conf_end_date\n| EVAL heap_util = COALESCE(heap_used_pct_p95, 100) / 100, disk_util = COALESCE(disk_used_pct_p95, 100) / 100, w_sum = conf_utilization_memory_weight + conf_utilization_storage_weight, capacity_utilization_score = CASE(w_sum > 0, (conf_utilization_memory_weight * heap_util + conf_utilization_storage_weight * disk_util) / w_sum, 1.0)\n| EVAL chargeable_pool = data_tier_capacity_ecu * COALESCE(capacity_utilization_score, 1.0), utilization_data_missing = heap_used_pct_p95 IS NULL AND disk_used_pct_p95 IS NULL\n| EVAL\n indexing = CASE (deployment_sum_indexing_time > 0, TO_DOUBLE(tier_and_datastream_sum_indexing_time) / deployment_sum_indexing_time * chargeable_pool),\n querying = CASE (deployment_sum_query_time > 0, TO_DOUBLE(tier_and_datastream_sum_query_time) / deployment_sum_query_time * chargeable_pool),\n data_set = CASE (deployment_sum_data_set_store_size > 0, TO_DOUBLE(tier_and_datastream_sum_data_set_store_size) / deployment_sum_data_set_store_size * chargeable_pool),\n store = CASE (deployment_sum_store_size > 0, TO_DOUBLE(tier_and_datastream_sum_store_size) / deployment_sum_store_size * chargeable_pool),\n storage = CASE (store == 0, data_set, store),\n total_weight_hot = conf_storage_weight + conf_query_weight + conf_indexing_weight,\n total_weight_cold = conf_storage_weight + conf_query_weight,\n blended = CASE (\n tier == \"hot/content\",\n TO_DOUBLE(((storage * conf_storage_weight) + (querying * conf_query_weight) + (indexing * conf_indexing_weight))) / total_weight_hot,\n TO_DOUBLE(((storage * conf_storage_weight) + (querying * conf_query_weight))) / total_weight_cold\n ) * COALESCE(conf_chargeable_unit_rate, conf_ecu_rate)\n| STATS\n agg_blended = sum(blended)\n BY\n datastream\n| WHERE agg_blended > 0\n| SORT agg_blended DESC\n| limit 20" - }, - "timeField": "@timestamp" - } - } - } + { + "columnId": "7ff73adf-6069-4878-af96-89acc86b9fb2", + "fieldName": "Storage", + "label": "Storage", + "meta": { + "esType": "double", + "params": { + "id": "number" }, - "filters": [], - "needsRefresh": false, - "query": { - "esql": "FROM billing_realized_pool_lookup\n| LOOKUP JOIN cluster_capacity_utilization_lookup ON composite_key\n| LOOKUP JOIN cluster_deployment_contribution_lookup ON composite_key\n| LOOKUP JOIN cluster_tier_and_datastream_contribution_lookup ON composite_key\n| LOOKUP JOIN chargeback_conf_lookup ON @timestamp >= conf_start_date AND @timestamp <= conf_end_date\n| EVAL heap_util = COALESCE(heap_used_pct_p95, 100) / 100, disk_util = COALESCE(disk_used_pct_p95, 100) / 100, w_sum = conf_utilization_memory_weight + conf_utilization_storage_weight, capacity_utilization_score = CASE(w_sum > 0, (conf_utilization_memory_weight * heap_util + conf_utilization_storage_weight * disk_util) / w_sum, 1.0)\n| EVAL chargeable_pool = data_tier_capacity_ecu * COALESCE(capacity_utilization_score, 1.0), utilization_data_missing = heap_used_pct_p95 IS NULL AND disk_used_pct_p95 IS NULL\n| EVAL\n indexing = CASE (deployment_sum_indexing_time > 0, TO_DOUBLE(tier_and_datastream_sum_indexing_time) / deployment_sum_indexing_time * chargeable_pool),\n querying = CASE (deployment_sum_query_time > 0, TO_DOUBLE(tier_and_datastream_sum_query_time) / deployment_sum_query_time * chargeable_pool),\n data_set = CASE (deployment_sum_data_set_store_size > 0, TO_DOUBLE(tier_and_datastream_sum_data_set_store_size) / deployment_sum_data_set_store_size * chargeable_pool),\n store = CASE (deployment_sum_store_size > 0, TO_DOUBLE(tier_and_datastream_sum_store_size) / deployment_sum_store_size * chargeable_pool),\n storage = CASE (store == 0, data_set, store),\n total_weight_hot = conf_storage_weight + conf_query_weight + conf_indexing_weight,\n total_weight_cold = conf_storage_weight + conf_query_weight,\n blended = CASE (\n tier == \"hot/content\",\n TO_DOUBLE(((storage * conf_storage_weight) + (querying * conf_query_weight) + (indexing * conf_indexing_weight))) / total_weight_hot,\n TO_DOUBLE(((storage * conf_storage_weight) + (querying * conf_query_weight))) / total_weight_cold\n ) * COALESCE(conf_chargeable_unit_rate, conf_ecu_rate)\n| STATS\n agg_blended = sum(blended)\n BY\n datastream\n| WHERE agg_blended > 0\n| SORT agg_blended DESC\n| limit 20" + "sourceParams": { + "indexPattern": "billing_realized_pool_lookup", + "sourceField": "Storage" }, - "visualization": { - "layers": [ - { - "categoryDisplay": "default", - "colorMapping": { - "assignments": [], - "colorMode": { - "type": "categorical" - }, - "paletteId": "default", - "specialAssignments": [ - { - "color": { - "type": "loop" - }, - "rules": [ - { - "type": "other" - } - ], - "touched": false - } - ] - }, - "layerId": "ee58e17d-eba7-48a7-bae8-2df3607d9653", - "layerType": "data", - "legendDisplay": "default", - "metrics": [ - "agg_blended" - ], - "nestedLegend": false, - "numberDisplay": "percent", - "primaryGroups": [ - "datastream" - ] - } - ], - "shape": "pie" - } + "type": "number" + } }, - "title": "Blended Cost by Data Stream", - "version": 2, - "visualizationType": "lnsPie" - }, - "title": "Blended Cost by Data Stream" - }, - "gridData": { - "h": 15, - "i": "e57765c6-40be-41cf-b3d2-e65ee8a51b67", - "sectionId": "d11ae80b-d7bb-474f-9082-81752ab2865d", - "w": 12, - "x": 36, - "y": 16 - }, - "panelIndex": "e57765c6-40be-41cf-b3d2-e65ee8a51b67", - "type": "lens" - }, - { - "embeddableConfig": { - "attributes": { - "references": [], - "state": { - "datasourceStates": { - "textBased": { - "indexPatternRefs": [], - "layers": { - "23693fb0-43a5-4f95-bd77-d6a54e18e34a": { - "allColumns": [ - { - "columnId": "Data tier", - "customLabel": true, - "fieldName": "Data tier", - "inMetricDimension": true, - "label": "Data Tier", - "meta": { - "esType": "keyword", - "type": "string" - } - }, - { - "columnId": "Indexing", - "customLabel": false, - "fieldName": "Indexing", - "inMetricDimension": true, - "label": "Indexing", - "meta": { - "esType": "double", - "type": "number" - } - }, - { - "columnId": "Querying", - "customLabel": false, - "fieldName": "Querying", - "inMetricDimension": true, - "label": "Querying", - "meta": { - "esType": "double", - "type": "number" - } - }, - { - "columnId": "Storage", - "customLabel": false, - "fieldName": "Storage", - "inMetricDimension": true, - "label": "Storage", - "meta": { - "esType": "double", - "type": "number" - } - }, - { - "columnId": "ce5b4b7d-63cc-4961-9d04-657a81926514", - "fieldName": "Data stream", - "meta": { - "esType": "keyword", - "type": "string" - } - }, - { - "columnId": "Data stream", - "fieldName": "Data stream", - "label": "Data stream", - "meta": { - "esType": "keyword", - "type": "string" - } - } - ], - "columns": [ - { - "columnId": "ce5b4b7d-63cc-4961-9d04-657a81926514", - "fieldName": "Data stream", - "meta": { - "esType": "keyword", - "type": "string" - } - }, - { - "columnId": "3433219b-e0b9-4ee9-bf82-e2d00fc614ed", - "fieldName": "Data tier", - "meta": { - "esType": "keyword", - "type": "string" - } - }, - { - "columnId": "f649e7a2-4d2c-45a3-aa64-99f2e29c2dd0", - "fieldName": "Indexing", - "label": "Indexing", - "meta": { - "esType": "double", - "params": { - "id": "number" - }, - "sourceParams": { - "indexPattern": "billing_realized_pool_lookup", - "sourceField": "Indexing" - }, - "type": "number" - } - }, - { - "columnId": "71737902-035d-4164-978e-2614106901ea", - "fieldName": "Querying", - "label": "Querying", - "meta": { - "esType": "double", - "params": { - "id": "number" - }, - "sourceParams": { - "indexPattern": "billing_realized_pool_lookup", - "sourceField": "Querying" - }, - "type": "number" - } - }, - { - "columnId": "7ff73adf-6069-4878-af96-89acc86b9fb2", - "fieldName": "Storage", - "label": "Storage", - "meta": { - "esType": "double", - "params": { - "id": "number" - }, - "sourceParams": { - "indexPattern": "billing_realized_pool_lookup", - "sourceField": "Storage" - }, - "type": "number" - } - }, - { - "columnId": "cbb5c7e4-d1f7-47a1-8c0a-766ac93d4362", - "fieldName": "Blended", - "label": "Blended", - "meta": { - "esType": "double", - "params": { - "id": "number" - }, - "sourceParams": { - "indexPattern": "billing_realized_pool_lookup", - "sourceField": "Blended" - }, - "type": "number" - } - } - ], - "index": "chargeback_integration", - "query": { - "esql": "FROM billing_realized_pool_lookup\n| LOOKUP JOIN cluster_capacity_utilization_lookup ON composite_key\n| LOOKUP JOIN cluster_deployment_contribution_lookup ON composite_key\n| LOOKUP JOIN chargeback_conf_lookup ON @timestamp >= conf_start_date AND @timestamp <= conf_end_date\n| EVAL heap_util = COALESCE(heap_used_pct_p95, 100) / 100, disk_util = COALESCE(disk_used_pct_p95, 100) / 100, w_sum = conf_utilization_memory_weight + conf_utilization_storage_weight, capacity_utilization_score = CASE(w_sum > 0, (conf_utilization_memory_weight * heap_util + conf_utilization_storage_weight * disk_util) / w_sum, 1.0)\n| EVAL chargeable_pool = data_tier_capacity_ecu * COALESCE(capacity_utilization_score, 1.0), utilization_data_missing = heap_used_pct_p95 IS NULL AND disk_used_pct_p95 IS NULL\n| LOOKUP JOIN cluster_tier_and_datastream_contribution_lookup ON composite_key\n| EVAL\n indexing = CASE(deployment_sum_indexing_time > 0, TO_DOUBLE(tier_and_datastream_sum_indexing_time) / deployment_sum_indexing_time * chargeable_pool) * COALESCE(conf_chargeable_unit_rate, conf_ecu_rate),\n querying = CASE(deployment_sum_query_time > 0, TO_DOUBLE(tier_and_datastream_sum_query_time) / deployment_sum_query_time * chargeable_pool) * COALESCE(conf_chargeable_unit_rate, conf_ecu_rate),\n data_set = CASE(deployment_sum_data_set_store_size > 0, TO_DOUBLE(tier_and_datastream_sum_data_set_store_size) / deployment_sum_data_set_store_size * chargeable_pool),\n store = CASE(deployment_sum_store_size > 0, TO_DOUBLE(tier_and_datastream_sum_store_size) / deployment_sum_store_size * chargeable_pool),\n storage = CASE(store == 0, data_set, store) * COALESCE(conf_chargeable_unit_rate, conf_ecu_rate),\n total_weight_hot = conf_storage_weight + conf_query_weight + conf_indexing_weight,\n total_weight_cold = conf_storage_weight + conf_query_weight,\n blended = CASE(\n tier == \"hot/content\",\n ((storage / COALESCE(conf_chargeable_unit_rate, conf_ecu_rate) * conf_storage_weight) + (querying / COALESCE(conf_chargeable_unit_rate, conf_ecu_rate) * conf_query_weight) + (indexing / COALESCE(conf_chargeable_unit_rate, conf_ecu_rate) * conf_indexing_weight)) / total_weight_hot,\n ((storage / COALESCE(conf_chargeable_unit_rate, conf_ecu_rate) * conf_storage_weight) + (querying / COALESCE(conf_chargeable_unit_rate, conf_ecu_rate) * conf_query_weight)) / total_weight_cold\n ) * COALESCE(conf_chargeable_unit_rate, conf_ecu_rate)\n| STATS\n Indexing = round(sum(indexing),2),\n Querying = round(sum(querying),2),\n Storage = round(sum(storage),2),\n Blended = round(sum(blended),2)\n BY datastream, tier\n| SORT Blended DESC\n| RENAME tier as `Data tier`, datastream as `Data stream`\n| Keep `Data stream`, `Data tier`, Indexing, Querying, Storage, Blended\n| WHERE `Data stream` is not null" - }, - "timeField": "@timestamp" - } - } - } + { + "columnId": "cbb5c7e4-d1f7-47a1-8c0a-766ac93d4362", + "fieldName": "Blended", + "label": "Blended", + "meta": { + "esType": "double", + "params": { + "id": "number" }, - "filters": [], - "needsRefresh": false, - "query": { - "esql": "FROM billing_realized_pool_lookup\n| LOOKUP JOIN cluster_capacity_utilization_lookup ON composite_key\n| LOOKUP JOIN cluster_deployment_contribution_lookup ON composite_key\n| LOOKUP JOIN chargeback_conf_lookup ON @timestamp >= conf_start_date AND @timestamp <= conf_end_date\n| EVAL heap_util = COALESCE(heap_used_pct_p95, 100) / 100, disk_util = COALESCE(disk_used_pct_p95, 100) / 100, w_sum = conf_utilization_memory_weight + conf_utilization_storage_weight, capacity_utilization_score = CASE(w_sum > 0, (conf_utilization_memory_weight * heap_util + conf_utilization_storage_weight * disk_util) / w_sum, 1.0)\n| EVAL chargeable_pool = data_tier_capacity_ecu * COALESCE(capacity_utilization_score, 1.0), utilization_data_missing = heap_used_pct_p95 IS NULL AND disk_used_pct_p95 IS NULL\n| LOOKUP JOIN cluster_tier_and_datastream_contribution_lookup ON composite_key\n| EVAL\n indexing = CASE(deployment_sum_indexing_time > 0, TO_DOUBLE(tier_and_datastream_sum_indexing_time) / deployment_sum_indexing_time * chargeable_pool) * COALESCE(conf_chargeable_unit_rate, conf_ecu_rate),\n querying = CASE(deployment_sum_query_time > 0, TO_DOUBLE(tier_and_datastream_sum_query_time) / deployment_sum_query_time * chargeable_pool) * COALESCE(conf_chargeable_unit_rate, conf_ecu_rate),\n data_set = CASE(deployment_sum_data_set_store_size > 0, TO_DOUBLE(tier_and_datastream_sum_data_set_store_size) / deployment_sum_data_set_store_size * chargeable_pool),\n store = CASE(deployment_sum_store_size > 0, TO_DOUBLE(tier_and_datastream_sum_store_size) / deployment_sum_store_size * chargeable_pool),\n storage = CASE(store == 0, data_set, store) * COALESCE(conf_chargeable_unit_rate, conf_ecu_rate),\n total_weight_hot = conf_storage_weight + conf_query_weight + conf_indexing_weight,\n total_weight_cold = conf_storage_weight + conf_query_weight,\n blended = CASE(\n tier == \"hot/content\",\n ((storage / COALESCE(conf_chargeable_unit_rate, conf_ecu_rate) * conf_storage_weight) + (querying / COALESCE(conf_chargeable_unit_rate, conf_ecu_rate) * conf_query_weight) + (indexing / COALESCE(conf_chargeable_unit_rate, conf_ecu_rate) * conf_indexing_weight)) / total_weight_hot,\n ((storage / COALESCE(conf_chargeable_unit_rate, conf_ecu_rate) * conf_storage_weight) + (querying / COALESCE(conf_chargeable_unit_rate, conf_ecu_rate) * conf_query_weight)) / total_weight_cold\n ) * COALESCE(conf_chargeable_unit_rate, conf_ecu_rate)\n| STATS\n Indexing = round(sum(indexing),2),\n Querying = round(sum(querying),2),\n Storage = round(sum(storage),2),\n Blended = round(sum(blended),2)\n BY datastream, tier\n| SORT Blended DESC\n| RENAME tier as `Data tier`, datastream as `Data stream`\n| Keep `Data stream`, `Data tier`, Indexing, Querying, Storage, Blended\n| WHERE `Data stream` is not null" + "sourceParams": { + "indexPattern": "billing_realized_pool_lookup", + "sourceField": "Blended" }, - "visualization": { - "columns": [ - { - "columnId": "ce5b4b7d-63cc-4961-9d04-657a81926514", - "isMetric": false, - "isTransposed": false - }, - { - "columnId": "3433219b-e0b9-4ee9-bf82-e2d00fc614ed", - "isMetric": false, - "isTransposed": false - }, - { - "columnId": "f649e7a2-4d2c-45a3-aa64-99f2e29c2dd0", - "isMetric": true, - "isTransposed": false - }, - { - "columnId": "71737902-035d-4164-978e-2614106901ea", - "isMetric": true, - "isTransposed": false - }, - { - "columnId": "7ff73adf-6069-4878-af96-89acc86b9fb2", - "isMetric": true, - "isTransposed": false - }, - { - "columnId": "cbb5c7e4-d1f7-47a1-8c0a-766ac93d4362", - "isMetric": true, - "isTransposed": false - } - ], - "layerId": "23693fb0-43a5-4f95-bd77-d6a54e18e34a", - "layerType": "data" - } - }, - "title": "Workload Breakdown by Data Tier", - "version": 2, - "visualizationType": "lnsDatatable" - }, - "enhancements": { - "dynamicActions": { - "events": [] + "type": "number" + } } - }, - "filters": [], - "query": { + ], + "index": "chargeback_integration", + "query": { "esql": "FROM billing_realized_pool_lookup\n| LOOKUP JOIN cluster_capacity_utilization_lookup ON composite_key\n| LOOKUP JOIN cluster_deployment_contribution_lookup ON composite_key\n| LOOKUP JOIN chargeback_conf_lookup ON @timestamp >= conf_start_date AND @timestamp <= conf_end_date\n| EVAL heap_util = COALESCE(heap_used_pct_p95, 100) / 100, disk_util = COALESCE(disk_used_pct_p95, 100) / 100, w_sum = conf_utilization_memory_weight + conf_utilization_storage_weight, capacity_utilization_score = CASE(w_sum > 0, (conf_utilization_memory_weight * heap_util + conf_utilization_storage_weight * disk_util) / w_sum, 1.0)\n| EVAL chargeable_pool = data_tier_capacity_ecu * COALESCE(capacity_utilization_score, 1.0), utilization_data_missing = heap_used_pct_p95 IS NULL AND disk_used_pct_p95 IS NULL\n| LOOKUP JOIN cluster_tier_and_datastream_contribution_lookup ON composite_key\n| EVAL\n indexing = CASE(deployment_sum_indexing_time > 0, TO_DOUBLE(tier_and_datastream_sum_indexing_time) / deployment_sum_indexing_time * chargeable_pool) * COALESCE(conf_chargeable_unit_rate, conf_ecu_rate),\n querying = CASE(deployment_sum_query_time > 0, TO_DOUBLE(tier_and_datastream_sum_query_time) / deployment_sum_query_time * chargeable_pool) * COALESCE(conf_chargeable_unit_rate, conf_ecu_rate),\n data_set = CASE(deployment_sum_data_set_store_size > 0, TO_DOUBLE(tier_and_datastream_sum_data_set_store_size) / deployment_sum_data_set_store_size * chargeable_pool),\n store = CASE(deployment_sum_store_size > 0, TO_DOUBLE(tier_and_datastream_sum_store_size) / deployment_sum_store_size * chargeable_pool),\n storage = CASE(store == 0, data_set, store) * COALESCE(conf_chargeable_unit_rate, conf_ecu_rate),\n total_weight_hot = conf_storage_weight + conf_query_weight + conf_indexing_weight,\n total_weight_cold = conf_storage_weight + conf_query_weight,\n blended = CASE(\n tier == \"hot/content\",\n ((storage / COALESCE(conf_chargeable_unit_rate, conf_ecu_rate) * conf_storage_weight) + (querying / COALESCE(conf_chargeable_unit_rate, conf_ecu_rate) * conf_query_weight) + (indexing / COALESCE(conf_chargeable_unit_rate, conf_ecu_rate) * conf_indexing_weight)) / total_weight_hot,\n ((storage / COALESCE(conf_chargeable_unit_rate, conf_ecu_rate) * conf_storage_weight) + (querying / COALESCE(conf_chargeable_unit_rate, conf_ecu_rate) * conf_query_weight)) / total_weight_cold\n ) * COALESCE(conf_chargeable_unit_rate, conf_ecu_rate)\n| STATS\n Indexing = round(sum(indexing),2),\n Querying = round(sum(querying),2),\n Storage = round(sum(storage),2),\n Blended = round(sum(blended),2)\n BY datastream, tier\n| SORT Blended DESC\n| RENAME tier as `Data tier`, datastream as `Data stream`\n| Keep `Data stream`, `Data tier`, Indexing, Querying, Storage, Blended\n| WHERE `Data stream` is not null" - }, - "syncColors": false, - "syncCursor": true, - "syncTooltips": false, - "title": "Workload Breakdown by Data Tier" - }, - "gridData": { - "h": 15, - "i": "7766db54-cadf-46f1-8417-ebe8267a3c23", - "sectionId": "d11ae80b-d7bb-474f-9082-81752ab2865d", - "w": 48, - "x": 0, - "y": 31 - }, - "panelIndex": "7766db54-cadf-46f1-8417-ebe8267a3c23", - "type": "lens" + }, + "timeField": "@timestamp" + } + } + } + }, + "filters": [], + "needsRefresh": false, + "query": { + "esql": "FROM billing_realized_pool_lookup\n| LOOKUP JOIN cluster_capacity_utilization_lookup ON composite_key\n| LOOKUP JOIN cluster_deployment_contribution_lookup ON composite_key\n| LOOKUP JOIN chargeback_conf_lookup ON @timestamp >= conf_start_date AND @timestamp <= conf_end_date\n| EVAL heap_util = COALESCE(heap_used_pct_p95, 100) / 100, disk_util = COALESCE(disk_used_pct_p95, 100) / 100, w_sum = conf_utilization_memory_weight + conf_utilization_storage_weight, capacity_utilization_score = CASE(w_sum > 0, (conf_utilization_memory_weight * heap_util + conf_utilization_storage_weight * disk_util) / w_sum, 1.0)\n| EVAL chargeable_pool = data_tier_capacity_ecu * COALESCE(capacity_utilization_score, 1.0), utilization_data_missing = heap_used_pct_p95 IS NULL AND disk_used_pct_p95 IS NULL\n| LOOKUP JOIN cluster_tier_and_datastream_contribution_lookup ON composite_key\n| EVAL\n indexing = CASE(deployment_sum_indexing_time > 0, TO_DOUBLE(tier_and_datastream_sum_indexing_time) / deployment_sum_indexing_time * chargeable_pool) * COALESCE(conf_chargeable_unit_rate, conf_ecu_rate),\n querying = CASE(deployment_sum_query_time > 0, TO_DOUBLE(tier_and_datastream_sum_query_time) / deployment_sum_query_time * chargeable_pool) * COALESCE(conf_chargeable_unit_rate, conf_ecu_rate),\n data_set = CASE(deployment_sum_data_set_store_size > 0, TO_DOUBLE(tier_and_datastream_sum_data_set_store_size) / deployment_sum_data_set_store_size * chargeable_pool),\n store = CASE(deployment_sum_store_size > 0, TO_DOUBLE(tier_and_datastream_sum_store_size) / deployment_sum_store_size * chargeable_pool),\n storage = CASE(store == 0, data_set, store) * COALESCE(conf_chargeable_unit_rate, conf_ecu_rate),\n total_weight_hot = conf_storage_weight + conf_query_weight + conf_indexing_weight,\n total_weight_cold = conf_storage_weight + conf_query_weight,\n blended = CASE(\n tier == \"hot/content\",\n ((storage / COALESCE(conf_chargeable_unit_rate, conf_ecu_rate) * conf_storage_weight) + (querying / COALESCE(conf_chargeable_unit_rate, conf_ecu_rate) * conf_query_weight) + (indexing / COALESCE(conf_chargeable_unit_rate, conf_ecu_rate) * conf_indexing_weight)) / total_weight_hot,\n ((storage / COALESCE(conf_chargeable_unit_rate, conf_ecu_rate) * conf_storage_weight) + (querying / COALESCE(conf_chargeable_unit_rate, conf_ecu_rate) * conf_query_weight)) / total_weight_cold\n ) * COALESCE(conf_chargeable_unit_rate, conf_ecu_rate)\n| STATS\n Indexing = round(sum(indexing),2),\n Querying = round(sum(querying),2),\n Storage = round(sum(storage),2),\n Blended = round(sum(blended),2)\n BY datastream, tier\n| SORT Blended DESC\n| RENAME tier as `Data tier`, datastream as `Data stream`\n| Keep `Data stream`, `Data tier`, Indexing, Querying, Storage, Blended\n| WHERE `Data stream` is not null" + }, + "visualization": { + "columns": [ + { + "columnId": "ce5b4b7d-63cc-4961-9d04-657a81926514", + "isMetric": false, + "isTransposed": false + }, + { + "columnId": "3433219b-e0b9-4ee9-bf82-e2d00fc614ed", + "isMetric": false, + "isTransposed": false + }, + { + "columnId": "f649e7a2-4d2c-45a3-aa64-99f2e29c2dd0", + "isMetric": true, + "isTransposed": false + }, + { + "columnId": "71737902-035d-4164-978e-2614106901ea", + "isMetric": true, + "isTransposed": false + }, + { + "columnId": "7ff73adf-6069-4878-af96-89acc86b9fb2", + "isMetric": true, + "isTransposed": false + }, + { + "columnId": "cbb5c7e4-d1f7-47a1-8c0a-766ac93d4362", + "isMetric": true, + "isTransposed": false + } + ], + "layerId": "23693fb0-43a5-4f95-bd77-d6a54e18e34a", + "layerType": "data" + } }, - { - "embeddableConfig": { - "enhancements": { - "dynamicActions": { - "events": [] - } - }, - "savedVis": { - "data": { - "aggs": [], - "searchSource": { - "filter": [], - "query": { - "language": "kuery", - "query": "" - } + "title": "Workload Breakdown by Data Tier", + "version": 2, + "visualizationType": "lnsDatatable" + }, + "enhancements": { + "dynamicActions": { + "events": [] + } + }, + "filters": [], + "query": { + "esql": "FROM billing_realized_pool_lookup\n| LOOKUP JOIN cluster_capacity_utilization_lookup ON composite_key\n| LOOKUP JOIN cluster_deployment_contribution_lookup ON composite_key\n| LOOKUP JOIN chargeback_conf_lookup ON @timestamp >= conf_start_date AND @timestamp <= conf_end_date\n| EVAL heap_util = COALESCE(heap_used_pct_p95, 100) / 100, disk_util = COALESCE(disk_used_pct_p95, 100) / 100, w_sum = conf_utilization_memory_weight + conf_utilization_storage_weight, capacity_utilization_score = CASE(w_sum > 0, (conf_utilization_memory_weight * heap_util + conf_utilization_storage_weight * disk_util) / w_sum, 1.0)\n| EVAL chargeable_pool = data_tier_capacity_ecu * COALESCE(capacity_utilization_score, 1.0), utilization_data_missing = heap_used_pct_p95 IS NULL AND disk_used_pct_p95 IS NULL\n| LOOKUP JOIN cluster_tier_and_datastream_contribution_lookup ON composite_key\n| EVAL\n indexing = CASE(deployment_sum_indexing_time > 0, TO_DOUBLE(tier_and_datastream_sum_indexing_time) / deployment_sum_indexing_time * chargeable_pool) * COALESCE(conf_chargeable_unit_rate, conf_ecu_rate),\n querying = CASE(deployment_sum_query_time > 0, TO_DOUBLE(tier_and_datastream_sum_query_time) / deployment_sum_query_time * chargeable_pool) * COALESCE(conf_chargeable_unit_rate, conf_ecu_rate),\n data_set = CASE(deployment_sum_data_set_store_size > 0, TO_DOUBLE(tier_and_datastream_sum_data_set_store_size) / deployment_sum_data_set_store_size * chargeable_pool),\n store = CASE(deployment_sum_store_size > 0, TO_DOUBLE(tier_and_datastream_sum_store_size) / deployment_sum_store_size * chargeable_pool),\n storage = CASE(store == 0, data_set, store) * COALESCE(conf_chargeable_unit_rate, conf_ecu_rate),\n total_weight_hot = conf_storage_weight + conf_query_weight + conf_indexing_weight,\n total_weight_cold = conf_storage_weight + conf_query_weight,\n blended = CASE(\n tier == \"hot/content\",\n ((storage / COALESCE(conf_chargeable_unit_rate, conf_ecu_rate) * conf_storage_weight) + (querying / COALESCE(conf_chargeable_unit_rate, conf_ecu_rate) * conf_query_weight) + (indexing / COALESCE(conf_chargeable_unit_rate, conf_ecu_rate) * conf_indexing_weight)) / total_weight_hot,\n ((storage / COALESCE(conf_chargeable_unit_rate, conf_ecu_rate) * conf_storage_weight) + (querying / COALESCE(conf_chargeable_unit_rate, conf_ecu_rate) * conf_query_weight)) / total_weight_cold\n ) * COALESCE(conf_chargeable_unit_rate, conf_ecu_rate)\n| STATS\n Indexing = round(sum(indexing),2),\n Querying = round(sum(querying),2),\n Storage = round(sum(storage),2),\n Blended = round(sum(blended),2)\n BY datastream, tier\n| SORT Blended DESC\n| RENAME tier as `Data tier`, datastream as `Data stream`\n| Keep `Data stream`, `Data tier`, Indexing, Querying, Storage, Blended\n| WHERE `Data stream` is not null" + }, + "syncColors": false, + "syncCursor": true, + "syncTooltips": false, + "title": "Workload Breakdown by Data Tier" + }, + "gridData": { + "h": 15, + "i": "7766db54-cadf-46f1-8417-ebe8267a3c23", + "sectionId": "d11ae80b-d7bb-474f-9082-81752ab2865d", + "w": 48, + "x": 0, + "y": 49 + }, + "panelIndex": "7766db54-cadf-46f1-8417-ebe8267a3c23", + "type": "lens" + }, + { + "embeddableConfig": { + "enhancements": { + "dynamicActions": { + "events": [] + } + }, + "savedVis": { + "data": { + "aggs": [], + "searchSource": { + "filter": [], + "query": { + "language": "kuery", + "query": "" + } + } + }, + "description": "", + "id": "", + "params": { + "fontSize": 12, + "markdown": "# Data tier and data stream overview", + "openLinksInNewTab": false + }, + "title": "", + "type": "markdown", + "uiState": {} + } + }, + "gridData": { + "h": 3, + "i": "a22d7ea0-bd5b-4372-a1bf-c3c44af10937", + "sectionId": "d11ae80b-d7bb-474f-9082-81752ab2865d", + "w": 48, + "x": 0, + "y": 0 + }, + "panelIndex": "a22d7ea0-bd5b-4372-a1bf-c3c44af10937", + "type": "visualization" + }, + { + "embeddableConfig": { + "attributes": { + "references": [], + "state": { + "datasourceStates": { + "textBased": { + "indexPatternRefs": [], + "layers": { + "0ee24895-1496-4b23-96ec-ec09a8f49332": { + "columns": [ + { + "columnId": "agg_indexing", + "customLabel": true, + "fieldName": "agg_indexing", + "inMetricDimension": true, + "label": "Normalized cost", + "meta": { + "esType": "double", + "type": "number" + }, + "params": { + "format": { + "id": "number", + "params": { + "decimals": 2 + } } + } }, - "description": "", - "id": "", - "params": { - "fontSize": 12, - "markdown": "# Data tier and data stream overview", - "openLinksInNewTab": false + { + "columnId": "@timestamp", + "customLabel": true, + "fieldName": "@timestamp", + "label": "Date", + "meta": { + "esType": "date", + "type": "date" + } }, - "title": "", - "type": "markdown", - "uiState": {} + { + "columnId": "datastream", + "customLabel": true, + "fieldName": "datastream", + "label": "Data stream", + "meta": { + "esType": "keyword", + "type": "string" + } + } + ], + "index": "chargeback_integration", + "query": { + "esql": "FROM billing_realized_pool_lookup\n| LOOKUP JOIN cluster_capacity_utilization_lookup ON composite_key\n| LOOKUP JOIN cluster_deployment_contribution_lookup ON composite_key\n| LOOKUP JOIN cluster_datastream_contribution_lookup ON composite_key\n| LOOKUP JOIN chargeback_conf_lookup ON @timestamp >= conf_start_date AND @timestamp <= conf_end_date\n| EVAL heap_util = COALESCE(heap_used_pct_p95, 100) / 100, disk_util = COALESCE(disk_used_pct_p95, 100) / 100, w_sum = conf_utilization_memory_weight + conf_utilization_storage_weight, capacity_utilization_score = CASE(w_sum > 0, (conf_utilization_memory_weight * heap_util + conf_utilization_storage_weight * disk_util) / w_sum, 1.0)\n| EVAL chargeable_pool = data_tier_capacity_ecu * COALESCE(capacity_utilization_score, 1.0), utilization_data_missing = heap_used_pct_p95 IS NULL AND disk_used_pct_p95 IS NULL\n| EVAL \n indexing = CASE (deployment_sum_indexing_time > 0, TO_DOUBLE(datastream_sum_indexing_time) / deployment_sum_indexing_time * chargeable_pool) * COALESCE(conf_chargeable_unit_rate, conf_ecu_rate)\n| STATS \n agg_indexing = sum(indexing)\n BY \n @timestamp, \n datastream\n| WHERE agg_indexing > 0\n| SORT agg_indexing DESC" + }, + "timeField": "@timestamp" } - }, - "gridData": { - "h": 3, - "i": "a22d7ea0-bd5b-4372-a1bf-c3c44af10937", - "sectionId": "d11ae80b-d7bb-474f-9082-81752ab2865d", - "w": 48, - "x": 0, - "y": 0 - }, - "panelIndex": "a22d7ea0-bd5b-4372-a1bf-c3c44af10937", - "type": "visualization" - }, - { - "embeddableConfig": { - "attributes": { - "references": [], - "state": { - "datasourceStates": { - "textBased": { - "indexPatternRefs": [], - "layers": { - "0ee24895-1496-4b23-96ec-ec09a8f49332": { - "columns": [ - { - "columnId": "agg_indexing", - "customLabel": true, - "fieldName": "agg_indexing", - "inMetricDimension": true, - "label": "Normalized cost", - "meta": { - "esType": "double", - "type": "number" - }, - "params": { - "format": { - "id": "number", - "params": { - "decimals": 2 - } - } - } - }, - { - "columnId": "@timestamp", - "customLabel": true, - "fieldName": "@timestamp", - "label": "Date", - "meta": { - "esType": "date", - "type": "date" - } - }, - { - "columnId": "datastream", - "customLabel": true, - "fieldName": "datastream", - "label": "Data stream", - "meta": { - "esType": "keyword", - "type": "string" - } - } - ], - "index": "chargeback_integration", - "query": { - "esql": "FROM billing_realized_pool_lookup\n| LOOKUP JOIN cluster_capacity_utilization_lookup ON composite_key\n| LOOKUP JOIN cluster_deployment_contribution_lookup ON composite_key\n| LOOKUP JOIN cluster_datastream_contribution_lookup ON composite_key\n| LOOKUP JOIN chargeback_conf_lookup ON @timestamp >= conf_start_date AND @timestamp <= conf_end_date\n| EVAL heap_util = COALESCE(heap_used_pct_p95, 100) / 100, disk_util = COALESCE(disk_used_pct_p95, 100) / 100, w_sum = conf_utilization_memory_weight + conf_utilization_storage_weight, capacity_utilization_score = CASE(w_sum > 0, (conf_utilization_memory_weight * heap_util + conf_utilization_storage_weight * disk_util) / w_sum, 1.0)\n| EVAL chargeable_pool = data_tier_capacity_ecu * COALESCE(capacity_utilization_score, 1.0), utilization_data_missing = heap_used_pct_p95 IS NULL AND disk_used_pct_p95 IS NULL\n| EVAL \n indexing = CASE (deployment_sum_indexing_time > 0, TO_DOUBLE(datastream_sum_indexing_time) / deployment_sum_indexing_time * chargeable_pool) * COALESCE(conf_chargeable_unit_rate, conf_ecu_rate)\n| STATS \n agg_indexing = sum(indexing)\n BY \n @timestamp, \n datastream\n| WHERE agg_indexing > 0\n| SORT agg_indexing DESC" - }, - "timeField": "@timestamp" - } - } - } - }, - "filters": [], - "needsRefresh": false, - "query": { - "esql": "FROM billing_realized_pool_lookup\n| LOOKUP JOIN cluster_capacity_utilization_lookup ON composite_key\n| LOOKUP JOIN cluster_deployment_contribution_lookup ON composite_key\n| LOOKUP JOIN cluster_datastream_contribution_lookup ON composite_key\n| LOOKUP JOIN chargeback_conf_lookup ON @timestamp >= conf_start_date AND @timestamp <= conf_end_date\n| EVAL heap_util = COALESCE(heap_used_pct_p95, 100) / 100, disk_util = COALESCE(disk_used_pct_p95, 100) / 100, w_sum = conf_utilization_memory_weight + conf_utilization_storage_weight, capacity_utilization_score = CASE(w_sum > 0, (conf_utilization_memory_weight * heap_util + conf_utilization_storage_weight * disk_util) / w_sum, 1.0)\n| EVAL chargeable_pool = data_tier_capacity_ecu * COALESCE(capacity_utilization_score, 1.0), utilization_data_missing = heap_used_pct_p95 IS NULL AND disk_used_pct_p95 IS NULL\n| EVAL \n indexing = CASE (deployment_sum_indexing_time > 0, TO_DOUBLE(datastream_sum_indexing_time) / deployment_sum_indexing_time * chargeable_pool) * COALESCE(conf_chargeable_unit_rate, conf_ecu_rate)\n| STATS \n agg_indexing = sum(indexing)\n BY \n @timestamp, \n datastream\n| WHERE agg_indexing > 0\n| SORT agg_indexing DESC" - }, - "visualization": { - "axisTitlesVisibilitySettings": { - "x": true, - "yLeft": true, - "yRight": true - }, - "fittingFunction": "Linear", - "gridlinesVisibilitySettings": { - "x": true, - "yLeft": true, - "yRight": true - }, - "labelsOrientation": { - "x": 0, - "yLeft": 0, - "yRight": 0 - }, - "layers": [ - { - "accessors": [ - "agg_indexing" - ], - "colorMapping": { - "assignments": [], - "colorMode": { - "type": "categorical" - }, - "paletteId": "eui_amsterdam_color_blind", - "specialAssignments": [ - { - "color": { - "type": "loop" - }, - "rules": [ - { - "type": "other" - } - ], - "touched": false - } - ] - }, - "layerId": "0ee24895-1496-4b23-96ec-ec09a8f49332", - "layerType": "data", - "seriesType": "bar_stacked", - "splitAccessor": "datastream", - "splitAccessors": [ - "datastream" - ], - "xAccessor": "@timestamp" - } - ], - "legend": { - "isVisible": false, - "position": "right", - "showSingleSeries": false - }, - "preferredSeriesType": "bar_stacked", - "tickLabelsVisibilitySettings": { - "x": true, - "yLeft": true, - "yRight": true - }, - "valueLabels": "hide" + } + } + }, + "filters": [], + "needsRefresh": false, + "query": { + "esql": "FROM billing_realized_pool_lookup\n| LOOKUP JOIN cluster_capacity_utilization_lookup ON composite_key\n| LOOKUP JOIN cluster_deployment_contribution_lookup ON composite_key\n| LOOKUP JOIN cluster_datastream_contribution_lookup ON composite_key\n| LOOKUP JOIN chargeback_conf_lookup ON @timestamp >= conf_start_date AND @timestamp <= conf_end_date\n| EVAL heap_util = COALESCE(heap_used_pct_p95, 100) / 100, disk_util = COALESCE(disk_used_pct_p95, 100) / 100, w_sum = conf_utilization_memory_weight + conf_utilization_storage_weight, capacity_utilization_score = CASE(w_sum > 0, (conf_utilization_memory_weight * heap_util + conf_utilization_storage_weight * disk_util) / w_sum, 1.0)\n| EVAL chargeable_pool = data_tier_capacity_ecu * COALESCE(capacity_utilization_score, 1.0), utilization_data_missing = heap_used_pct_p95 IS NULL AND disk_used_pct_p95 IS NULL\n| EVAL \n indexing = CASE (deployment_sum_indexing_time > 0, TO_DOUBLE(datastream_sum_indexing_time) / deployment_sum_indexing_time * chargeable_pool) * COALESCE(conf_chargeable_unit_rate, conf_ecu_rate)\n| STATS \n agg_indexing = sum(indexing)\n BY \n @timestamp, \n datastream\n| WHERE agg_indexing > 0\n| SORT agg_indexing DESC" + }, + "visualization": { + "axisTitlesVisibilitySettings": { + "x": true, + "yLeft": true, + "yRight": true + }, + "fittingFunction": "Linear", + "gridlinesVisibilitySettings": { + "x": true, + "yLeft": true, + "yRight": true + }, + "labelsOrientation": { + "x": 0, + "yLeft": 0, + "yRight": 0 + }, + "layers": [ + { + "accessors": [ + "agg_indexing" + ], + "colorMapping": { + "assignments": [], + "colorMode": { + "type": "categorical" + }, + "paletteId": "eui_amsterdam_color_blind", + "specialAssignments": [ + { + "color": { + "type": "loop" + }, + "rules": [ + { + "type": "other" } - }, - "title": "Indexing Cost by Data Stream over Time", - "version": 2, - "visualizationType": "lnsXY" + ], + "touched": false + } + ] }, - "title": "Indexing Cost by Data Stream over Time" - }, - "gridData": { - "h": 14, - "i": "a85aea2f-8e25-4bef-b09a-32f0a5cb33f9", - "sectionId": "7f7cc9b5-13b3-4962-98f6-5b18a1d8ca80", - "w": 16, - "x": 0, - "y": 6 - }, - "panelIndex": "a85aea2f-8e25-4bef-b09a-32f0a5cb33f9", - "type": "lens" + "layerId": "0ee24895-1496-4b23-96ec-ec09a8f49332", + "layerType": "data", + "seriesType": "bar_stacked", + "splitAccessor": "datastream", + "splitAccessors": [ + "datastream" + ], + "xAccessor": "@timestamp" + } + ], + "legend": { + "isVisible": false, + "position": "right", + "showSingleSeries": false + }, + "preferredSeriesType": "bar_stacked", + "tickLabelsVisibilitySettings": { + "x": true, + "yLeft": true, + "yRight": true + }, + "valueLabels": "hide" + } }, - { - "embeddableConfig": { - "attributes": { - "references": [], - "state": { - "datasourceStates": { - "textBased": { - "indexPatternRefs": [], - "layers": { - "d24af063-ec41-4736-bf4e-70f1fe4da4cf": { - "columns": [ - { - "columnId": "agg_indexing", - "customLabel": true, - "fieldName": "agg_indexing", - "inMetricDimension": true, - "label": "Normalized cost", - "meta": { - "esType": "double", - "type": "number" - }, - "params": { - "format": { - "id": "number", - "params": { - "decimals": 2 - } - } - } - }, - { - "columnId": "@timestamp", - "customLabel": true, - "fieldName": "@timestamp", - "label": "Date", - "meta": { - "esType": "date", - "type": "date" - } - }, - { - "columnId": "tier", - "customLabel": true, - "fieldName": "tier", - "label": "Data Tier", - "meta": { - "esType": "keyword", - "type": "string" - } - } - ], - "index": "chargeback_integration", - "query": { - "esql": "FROM billing_realized_pool_lookup\n| LOOKUP JOIN cluster_capacity_utilization_lookup ON composite_key\n| LOOKUP JOIN cluster_deployment_contribution_lookup ON composite_key\n| LOOKUP JOIN cluster_tier_contribution_lookup ON composite_key\n| LOOKUP JOIN chargeback_conf_lookup ON @timestamp >= conf_start_date AND @timestamp <= conf_end_date\n| EVAL heap_util = COALESCE(heap_used_pct_p95, 100) / 100, disk_util = COALESCE(disk_used_pct_p95, 100) / 100, w_sum = conf_utilization_memory_weight + conf_utilization_storage_weight, capacity_utilization_score = CASE(w_sum > 0, (conf_utilization_memory_weight * heap_util + conf_utilization_storage_weight * disk_util) / w_sum, 1.0)\n| EVAL chargeable_pool = data_tier_capacity_ecu * COALESCE(capacity_utilization_score, 1.0), utilization_data_missing = heap_used_pct_p95 IS NULL AND disk_used_pct_p95 IS NULL\n| EVAL \n indexing = CASE (\n deployment_sum_indexing_time > 0,\n TO_DOUBLE(tier_sum_indexing_time) / deployment_sum_indexing_time * chargeable_pool\n ) * COALESCE(conf_chargeable_unit_rate, conf_ecu_rate)\n| STATS \n agg_indexing = sum(indexing)\n BY \n @timestamp,\n tier\n| WHERE agg_indexing > 0" - }, - "timeField": "@timestamp" - } - } - } - }, - "filters": [], - "needsRefresh": false, - "query": { - "esql": "FROM billing_realized_pool_lookup\n| LOOKUP JOIN cluster_capacity_utilization_lookup ON composite_key\n| LOOKUP JOIN cluster_deployment_contribution_lookup ON composite_key\n| LOOKUP JOIN cluster_tier_contribution_lookup ON composite_key\n| LOOKUP JOIN chargeback_conf_lookup ON @timestamp >= conf_start_date AND @timestamp <= conf_end_date\n| EVAL heap_util = COALESCE(heap_used_pct_p95, 100) / 100, disk_util = COALESCE(disk_used_pct_p95, 100) / 100, w_sum = conf_utilization_memory_weight + conf_utilization_storage_weight, capacity_utilization_score = CASE(w_sum > 0, (conf_utilization_memory_weight * heap_util + conf_utilization_storage_weight * disk_util) / w_sum, 1.0)\n| EVAL chargeable_pool = data_tier_capacity_ecu * COALESCE(capacity_utilization_score, 1.0), utilization_data_missing = heap_used_pct_p95 IS NULL AND disk_used_pct_p95 IS NULL\n| EVAL \n indexing = CASE (\n deployment_sum_indexing_time > 0,\n TO_DOUBLE(tier_sum_indexing_time) / deployment_sum_indexing_time * chargeable_pool\n ) * COALESCE(conf_chargeable_unit_rate, conf_ecu_rate)\n| STATS \n agg_indexing = sum(indexing)\n BY \n @timestamp,\n tier\n| WHERE agg_indexing > 0" - }, - "visualization": { - "axisTitlesVisibilitySettings": { - "x": true, - "yLeft": true, - "yRight": true - }, - "fittingFunction": "Linear", - "gridlinesVisibilitySettings": { - "x": true, - "yLeft": true, - "yRight": true - }, - "labelsOrientation": { - "x": 0, - "yLeft": 0, - "yRight": 0 - }, - "layers": [ - { - "accessors": [ - "agg_indexing" - ], - "colorMapping": { - "assignments": [], - "colorMode": { - "type": "categorical" - }, - "paletteId": "eui_amsterdam_color_blind", - "specialAssignments": [ - { - "color": { - "type": "loop" - }, - "rules": [ - { - "type": "other" - } - ], - "touched": false - } - ] - }, - "layerId": "d24af063-ec41-4736-bf4e-70f1fe4da4cf", - "layerType": "data", - "seriesType": "bar_stacked", - "splitAccessor": "tier", - "splitAccessors": [ - "tier" - ], - "xAccessor": "@timestamp" - } - ], - "legend": { - "isVisible": false, - "position": "right", - "showSingleSeries": false - }, - "preferredSeriesType": "bar_stacked", - "tickLabelsVisibilitySettings": { - "x": true, - "yLeft": true, - "yRight": true - }, - "valueLabels": "hide" + "title": "Indexing Cost by Data Stream over Time", + "version": 2, + "visualizationType": "lnsXY" + }, + "title": "Indexing Cost by Data Stream over Time" + }, + "gridData": { + "h": 14, + "i": "a85aea2f-8e25-4bef-b09a-32f0a5cb33f9", + "sectionId": "7f7cc9b5-13b3-4962-98f6-5b18a1d8ca80", + "w": 16, + "x": 0, + "y": 6 + }, + "panelIndex": "a85aea2f-8e25-4bef-b09a-32f0a5cb33f9", + "type": "lens" + }, + { + "embeddableConfig": { + "attributes": { + "references": [], + "state": { + "datasourceStates": { + "textBased": { + "indexPatternRefs": [], + "layers": { + "d24af063-ec41-4736-bf4e-70f1fe4da4cf": { + "columns": [ + { + "columnId": "agg_indexing", + "customLabel": true, + "fieldName": "agg_indexing", + "inMetricDimension": true, + "label": "Normalized cost", + "meta": { + "esType": "double", + "type": "number" + }, + "params": { + "format": { + "id": "number", + "params": { + "decimals": 2 + } } + } }, - "title": "Indexing Cost by Tier over Time", - "version": 2, - "visualizationType": "lnsXY" - }, - "title": "Indexing Cost by Tier over Time" - }, - "gridData": { - "h": 14, - "i": "2f961fd2-5b04-4c4d-aafb-76af69b06130", - "sectionId": "7f7cc9b5-13b3-4962-98f6-5b18a1d8ca80", - "w": 16, - "x": 16, - "y": 6 - }, - "panelIndex": "2f961fd2-5b04-4c4d-aafb-76af69b06130", - "type": "lens" - }, - { - "embeddableConfig": { - "attributes": { - "references": [], - "state": { - "datasourceStates": { - "textBased": { - "indexPatternRefs": [], - "layers": { - "d5520b0e-567f-4f0a-b424-fc79eb79e2cc": { - "columns": [ - { - "columnId": "@timestamp", - "customLabel": true, - "fieldName": "@timestamp", - "label": "Date", - "meta": { - "esType": "date", - "type": "date" - } - }, - { - "columnId": "datastream", - "customLabel": true, - "fieldName": "datastream", - "label": "Data stream", - "meta": { - "esType": "keyword", - "type": "string" - } - }, - { - "columnId": "agg_indexing", - "customLabel": true, - "fieldName": "agg_indexing", - "inMetricDimension": true, - "label": "% Normalized cost", - "meta": { - "esType": "long", - "type": "number" - }, - "params": { - "format": { - "id": "percent", - "params": { - "decimals": 0 - } - } - } - } - ], - "index": "chargeback_integration", - "query": { - "esql": "FROM billing_realized_pool_lookup\n| LOOKUP JOIN cluster_capacity_utilization_lookup ON composite_key\n| LOOKUP JOIN cluster_deployment_contribution_lookup ON composite_key\n| LOOKUP JOIN cluster_datastream_contribution_lookup ON composite_key\n| LOOKUP JOIN chargeback_conf_lookup ON @timestamp >= conf_start_date AND @timestamp <= conf_end_date\n| EVAL heap_util = COALESCE(heap_used_pct_p95, 100) / 100, disk_util = COALESCE(disk_used_pct_p95, 100) / 100, w_sum = conf_utilization_memory_weight + conf_utilization_storage_weight, capacity_utilization_score = CASE(w_sum > 0, (conf_utilization_memory_weight * heap_util + conf_utilization_storage_weight * disk_util) / w_sum, 1.0)\n| EVAL chargeable_pool = data_tier_capacity_ecu * COALESCE(capacity_utilization_score, 1.0), utilization_data_missing = heap_used_pct_p95 IS NULL AND disk_used_pct_p95 IS NULL\n| EVAL \n indexing = CASE (\n deployment_sum_indexing_time > 0,\n TO_DOUBLE(datastream_sum_indexing_time) / deployment_sum_indexing_time * chargeable_pool\n ) * COALESCE(conf_chargeable_unit_rate, conf_ecu_rate)\n| STATS \n agg_indexing = sum(indexing)\n BY \n @timestamp,\n datastream\n| WHERE agg_indexing > 0\n| SORT agg_indexing DESC" - }, - "timeField": "@timestamp" - } - } - } - }, - "filters": [], - "needsRefresh": false, - "query": { - "esql": "FROM billing_realized_pool_lookup\n| LOOKUP JOIN cluster_capacity_utilization_lookup ON composite_key\n| LOOKUP JOIN cluster_deployment_contribution_lookup ON composite_key\n| LOOKUP JOIN cluster_datastream_contribution_lookup ON composite_key\n| LOOKUP JOIN chargeback_conf_lookup ON @timestamp >= conf_start_date AND @timestamp <= conf_end_date\n| EVAL heap_util = COALESCE(heap_used_pct_p95, 100) / 100, disk_util = COALESCE(disk_used_pct_p95, 100) / 100, w_sum = conf_utilization_memory_weight + conf_utilization_storage_weight, capacity_utilization_score = CASE(w_sum > 0, (conf_utilization_memory_weight * heap_util + conf_utilization_storage_weight * disk_util) / w_sum, 1.0)\n| EVAL chargeable_pool = data_tier_capacity_ecu * COALESCE(capacity_utilization_score, 1.0), utilization_data_missing = heap_used_pct_p95 IS NULL AND disk_used_pct_p95 IS NULL\n| EVAL \n indexing = CASE (\n deployment_sum_indexing_time > 0,\n TO_DOUBLE(datastream_sum_indexing_time) / deployment_sum_indexing_time * chargeable_pool\n ) * COALESCE(conf_chargeable_unit_rate, conf_ecu_rate)\n| STATS \n agg_indexing = sum(indexing)\n BY \n @timestamp,\n datastream\n| WHERE agg_indexing > 0\n| SORT agg_indexing DESC" - }, - "visualization": { - "axisTitlesVisibilitySettings": { - "x": true, - "yLeft": true, - "yRight": true - }, - "fittingFunction": "Linear", - "gridlinesVisibilitySettings": { - "x": true, - "yLeft": true, - "yRight": true - }, - "labelsOrientation": { - "x": 0, - "yLeft": 0, - "yRight": 0 - }, - "layers": [ - { - "accessors": [ - "agg_indexing" - ], - "colorMapping": { - "assignments": [], - "colorMode": { - "type": "categorical" - }, - "paletteId": "eui_amsterdam_color_blind", - "specialAssignments": [ - { - "color": { - "type": "loop" - }, - "rules": [ - { - "type": "other" - } - ], - "touched": false - } - ] - }, - "layerId": "d5520b0e-567f-4f0a-b424-fc79eb79e2cc", - "layerType": "data", - "seriesType": "bar_percentage_stacked", - "splitAccessor": "datastream", - "splitAccessors": [ - "datastream" - ], - "xAccessor": "@timestamp" - } - ], - "legend": { - "isVisible": false, - "position": "right", - "showSingleSeries": false - }, - "preferredSeriesType": "bar_stacked", - "tickLabelsVisibilitySettings": { - "x": true, - "yLeft": true, - "yRight": true - }, - "valueLabels": "hide" - } + { + "columnId": "@timestamp", + "customLabel": true, + "fieldName": "@timestamp", + "label": "Date", + "meta": { + "esType": "date", + "type": "date" + } }, - "title": "Indexing Cost Share by Data Stream over Time (%)", - "version": 2, - "visualizationType": "lnsXY" - }, - "title": "Indexing Cost Share by Data Stream over Time (%)" - }, - "gridData": { - "h": 14, - "i": "c895d51f-09ee-4624-8674-1d77a8d1df61", - "sectionId": "7f7cc9b5-13b3-4962-98f6-5b18a1d8ca80", - "w": 16, - "x": 32, - "y": 6 - }, - "panelIndex": "c895d51f-09ee-4624-8674-1d77a8d1df61", - "type": "lens" - }, - { - "embeddableConfig": { - "attributes": { - "references": [], - "state": { - "datasourceStates": { - "textBased": { - "indexPatternRefs": [], - "layers": { - "1a2f8269-f56f-4c02-8506-76b723acc828": { - "columns": [ - { - "columnId": "agg_querying", - "customLabel": true, - "fieldName": "agg_querying", - "inMetricDimension": true, - "label": "Normalized cost", - "meta": { - "esType": "double", - "type": "number" - }, - "params": { - "format": { - "id": "number", - "params": { - "decimals": 2 - } - } - } - }, - { - "columnId": "@timestamp", - "customLabel": true, - "fieldName": "@timestamp", - "label": "Date", - "meta": { - "esType": "date", - "type": "date" - } - }, - { - "columnId": "datastream", - "customLabel": true, - "fieldName": "datastream", - "label": "Data stream", - "meta": { - "esType": "keyword", - "type": "string" - } - } - ], - "index": "chargeback_integration", - "query": { - "esql": "FROM billing_realized_pool_lookup\n| LOOKUP JOIN cluster_capacity_utilization_lookup ON composite_key\n| LOOKUP JOIN cluster_deployment_contribution_lookup ON composite_key\n| LOOKUP JOIN cluster_datastream_contribution_lookup ON composite_key\n| LOOKUP JOIN chargeback_conf_lookup ON @timestamp >= conf_start_date AND @timestamp <= conf_end_date\n| EVAL heap_util = COALESCE(heap_used_pct_p95, 100) / 100, disk_util = COALESCE(disk_used_pct_p95, 100) / 100, w_sum = conf_utilization_memory_weight + conf_utilization_storage_weight, capacity_utilization_score = CASE(w_sum > 0, (conf_utilization_memory_weight * heap_util + conf_utilization_storage_weight * disk_util) / w_sum, 1.0)\n| EVAL chargeable_pool = data_tier_capacity_ecu * COALESCE(capacity_utilization_score, 1.0), utilization_data_missing = heap_used_pct_p95 IS NULL AND disk_used_pct_p95 IS NULL\n| EVAL \n querying = CASE (\n deployment_sum_query_time > 0,\n TO_DOUBLE(datastream_sum_query_time) / deployment_sum_query_time * chargeable_pool\n ) * COALESCE(conf_chargeable_unit_rate, conf_ecu_rate)\n| STATS \n agg_querying = sum(querying) \n BY \n @timestamp, \n datastream\n| WHERE agg_querying > 0\n| SORT agg_querying DESC" - }, - "timeField": "@timestamp" - } - } - } - }, - "filters": [], - "needsRefresh": false, - "query": { - "esql": "FROM billing_realized_pool_lookup\n| LOOKUP JOIN cluster_capacity_utilization_lookup ON composite_key\n| LOOKUP JOIN cluster_deployment_contribution_lookup ON composite_key\n| LOOKUP JOIN cluster_datastream_contribution_lookup ON composite_key\n| LOOKUP JOIN chargeback_conf_lookup ON @timestamp >= conf_start_date AND @timestamp <= conf_end_date\n| EVAL heap_util = COALESCE(heap_used_pct_p95, 100) / 100, disk_util = COALESCE(disk_used_pct_p95, 100) / 100, w_sum = conf_utilization_memory_weight + conf_utilization_storage_weight, capacity_utilization_score = CASE(w_sum > 0, (conf_utilization_memory_weight * heap_util + conf_utilization_storage_weight * disk_util) / w_sum, 1.0)\n| EVAL chargeable_pool = data_tier_capacity_ecu * COALESCE(capacity_utilization_score, 1.0), utilization_data_missing = heap_used_pct_p95 IS NULL AND disk_used_pct_p95 IS NULL\n| EVAL \n querying = CASE (\n deployment_sum_query_time > 0,\n TO_DOUBLE(datastream_sum_query_time) / deployment_sum_query_time * chargeable_pool\n ) * COALESCE(conf_chargeable_unit_rate, conf_ecu_rate)\n| STATS \n agg_querying = sum(querying) \n BY \n @timestamp, \n datastream\n| WHERE agg_querying > 0\n| SORT agg_querying DESC" - }, - "visualization": { - "axisTitlesVisibilitySettings": { - "x": true, - "yLeft": true, - "yRight": true - }, - "fittingFunction": "Linear", - "gridlinesVisibilitySettings": { - "x": true, - "yLeft": true, - "yRight": true - }, - "labelsOrientation": { - "x": 0, - "yLeft": 0, - "yRight": 0 - }, - "layers": [ - { - "accessors": [ - "agg_querying" - ], - "colorMapping": { - "assignments": [], - "colorMode": { - "type": "categorical" - }, - "paletteId": "eui_amsterdam_color_blind", - "specialAssignments": [ - { - "color": { - "type": "loop" - }, - "rules": [ - { - "type": "other" - } - ], - "touched": false - } - ] - }, - "layerId": "1a2f8269-f56f-4c02-8506-76b723acc828", - "layerType": "data", - "seriesType": "bar_stacked", - "splitAccessor": "datastream", - "splitAccessors": [ - "datastream" - ], - "xAccessor": "@timestamp" - } - ], - "legend": { - "isVisible": false, - "position": "right", - "showSingleSeries": false - }, - "preferredSeriesType": "bar_stacked", - "tickLabelsVisibilitySettings": { - "x": true, - "yLeft": true, - "yRight": true - }, - "valueLabels": "hide" + { + "columnId": "tier", + "customLabel": true, + "fieldName": "tier", + "label": "Data Tier", + "meta": { + "esType": "keyword", + "type": "string" + } + } + ], + "index": "chargeback_integration", + "query": { + "esql": "FROM billing_realized_pool_lookup\n| LOOKUP JOIN cluster_capacity_utilization_lookup ON composite_key\n| LOOKUP JOIN cluster_deployment_contribution_lookup ON composite_key\n| LOOKUP JOIN cluster_tier_contribution_lookup ON composite_key\n| LOOKUP JOIN chargeback_conf_lookup ON @timestamp >= conf_start_date AND @timestamp <= conf_end_date\n| EVAL heap_util = COALESCE(heap_used_pct_p95, 100) / 100, disk_util = COALESCE(disk_used_pct_p95, 100) / 100, w_sum = conf_utilization_memory_weight + conf_utilization_storage_weight, capacity_utilization_score = CASE(w_sum > 0, (conf_utilization_memory_weight * heap_util + conf_utilization_storage_weight * disk_util) / w_sum, 1.0)\n| EVAL chargeable_pool = data_tier_capacity_ecu * COALESCE(capacity_utilization_score, 1.0), utilization_data_missing = heap_used_pct_p95 IS NULL AND disk_used_pct_p95 IS NULL\n| EVAL \n indexing = CASE (\n deployment_sum_indexing_time > 0,\n TO_DOUBLE(tier_sum_indexing_time) / deployment_sum_indexing_time * chargeable_pool\n ) * COALESCE(conf_chargeable_unit_rate, conf_ecu_rate)\n| STATS \n agg_indexing = sum(indexing)\n BY \n @timestamp,\n tier\n| WHERE agg_indexing > 0" + }, + "timeField": "@timestamp" + } + } + } + }, + "filters": [], + "needsRefresh": false, + "query": { + "esql": "FROM billing_realized_pool_lookup\n| LOOKUP JOIN cluster_capacity_utilization_lookup ON composite_key\n| LOOKUP JOIN cluster_deployment_contribution_lookup ON composite_key\n| LOOKUP JOIN cluster_tier_contribution_lookup ON composite_key\n| LOOKUP JOIN chargeback_conf_lookup ON @timestamp >= conf_start_date AND @timestamp <= conf_end_date\n| EVAL heap_util = COALESCE(heap_used_pct_p95, 100) / 100, disk_util = COALESCE(disk_used_pct_p95, 100) / 100, w_sum = conf_utilization_memory_weight + conf_utilization_storage_weight, capacity_utilization_score = CASE(w_sum > 0, (conf_utilization_memory_weight * heap_util + conf_utilization_storage_weight * disk_util) / w_sum, 1.0)\n| EVAL chargeable_pool = data_tier_capacity_ecu * COALESCE(capacity_utilization_score, 1.0), utilization_data_missing = heap_used_pct_p95 IS NULL AND disk_used_pct_p95 IS NULL\n| EVAL \n indexing = CASE (\n deployment_sum_indexing_time > 0,\n TO_DOUBLE(tier_sum_indexing_time) / deployment_sum_indexing_time * chargeable_pool\n ) * COALESCE(conf_chargeable_unit_rate, conf_ecu_rate)\n| STATS \n agg_indexing = sum(indexing)\n BY \n @timestamp,\n tier\n| WHERE agg_indexing > 0" + }, + "visualization": { + "axisTitlesVisibilitySettings": { + "x": true, + "yLeft": true, + "yRight": true + }, + "fittingFunction": "Linear", + "gridlinesVisibilitySettings": { + "x": true, + "yLeft": true, + "yRight": true + }, + "labelsOrientation": { + "x": 0, + "yLeft": 0, + "yRight": 0 + }, + "layers": [ + { + "accessors": [ + "agg_indexing" + ], + "colorMapping": { + "assignments": [], + "colorMode": { + "type": "categorical" + }, + "paletteId": "eui_amsterdam_color_blind", + "specialAssignments": [ + { + "color": { + "type": "loop" + }, + "rules": [ + { + "type": "other" } - }, - "title": "Query Cost by Data Stream over Time", - "version": 2, - "visualizationType": "lnsXY" + ], + "touched": false + } + ] }, - "title": "Query Cost by Data Stream over Time" - }, - "gridData": { - "h": 14, - "i": "48aef73e-b6de-424a-85fa-74f46d9bded4", - "sectionId": "7f7cc9b5-13b3-4962-98f6-5b18a1d8ca80", - "w": 16, - "x": 0, - "y": 23 - }, - "panelIndex": "48aef73e-b6de-424a-85fa-74f46d9bded4", - "type": "lens" + "layerId": "d24af063-ec41-4736-bf4e-70f1fe4da4cf", + "layerType": "data", + "seriesType": "bar_stacked", + "splitAccessor": "tier", + "splitAccessors": [ + "tier" + ], + "xAccessor": "@timestamp" + } + ], + "legend": { + "isVisible": false, + "position": "right", + "showSingleSeries": false + }, + "preferredSeriesType": "bar_stacked", + "tickLabelsVisibilitySettings": { + "x": true, + "yLeft": true, + "yRight": true + }, + "valueLabels": "hide" + } }, - { - "embeddableConfig": { - "attributes": { - "references": [], - "state": { - "datasourceStates": { - "textBased": { - "indexPatternRefs": [], - "layers": { - "61842c44-afd9-429c-bf2e-18a4a079e5e8": { - "columns": [ - { - "columnId": "agg_querying", - "customLabel": true, - "fieldName": "agg_querying", - "inMetricDimension": true, - "label": "Normalized cost", - "meta": { - "esType": "double", - "type": "number" - }, - "params": { - "format": { - "id": "number", - "params": { - "decimals": 2 - } - } - } - }, - { - "columnId": "@timestamp", - "customLabel": true, - "fieldName": "@timestamp", - "label": "Date", - "meta": { - "esType": "date", - "type": "date" - } - }, - { - "columnId": "tier", - "customLabel": true, - "fieldName": "tier", - "label": "Data Tier", - "meta": { - "esType": "keyword", - "type": "string" - } - } - ], - "index": "chargeback_integration", - "query": { - "esql": "FROM billing_realized_pool_lookup\n| LOOKUP JOIN cluster_capacity_utilization_lookup ON composite_key\n| LOOKUP JOIN cluster_deployment_contribution_lookup ON composite_key\n| LOOKUP JOIN cluster_tier_contribution_lookup ON composite_key\n| LOOKUP JOIN chargeback_conf_lookup ON @timestamp >= conf_start_date AND @timestamp <= conf_end_date\n| EVAL heap_util = COALESCE(heap_used_pct_p95, 100) / 100, disk_util = COALESCE(disk_used_pct_p95, 100) / 100, w_sum = conf_utilization_memory_weight + conf_utilization_storage_weight, capacity_utilization_score = CASE(w_sum > 0, (conf_utilization_memory_weight * heap_util + conf_utilization_storage_weight * disk_util) / w_sum, 1.0)\n| EVAL chargeable_pool = data_tier_capacity_ecu * COALESCE(capacity_utilization_score, 1.0), utilization_data_missing = heap_used_pct_p95 IS NULL AND disk_used_pct_p95 IS NULL\n| EVAL \n querying = CASE (\n deployment_sum_query_time > 0,\n TO_DOUBLE(tier_sum_query_time) / deployment_sum_query_time * chargeable_pool\n ) * COALESCE(conf_chargeable_unit_rate, conf_ecu_rate)\n| STATS \n agg_querying = sum(querying)\n BY \n @timestamp,\n tier\n| WHERE agg_querying > 0" - }, - "timeField": "@timestamp" - } - } - } - }, - "filters": [], - "needsRefresh": false, - "query": { - "esql": "FROM billing_realized_pool_lookup\n| LOOKUP JOIN cluster_capacity_utilization_lookup ON composite_key\n| LOOKUP JOIN cluster_deployment_contribution_lookup ON composite_key\n| LOOKUP JOIN cluster_tier_contribution_lookup ON composite_key\n| LOOKUP JOIN chargeback_conf_lookup ON @timestamp >= conf_start_date AND @timestamp <= conf_end_date\n| EVAL heap_util = COALESCE(heap_used_pct_p95, 100) / 100, disk_util = COALESCE(disk_used_pct_p95, 100) / 100, w_sum = conf_utilization_memory_weight + conf_utilization_storage_weight, capacity_utilization_score = CASE(w_sum > 0, (conf_utilization_memory_weight * heap_util + conf_utilization_storage_weight * disk_util) / w_sum, 1.0)\n| EVAL chargeable_pool = data_tier_capacity_ecu * COALESCE(capacity_utilization_score, 1.0), utilization_data_missing = heap_used_pct_p95 IS NULL AND disk_used_pct_p95 IS NULL\n| EVAL \n querying = CASE (\n deployment_sum_query_time > 0,\n TO_DOUBLE(tier_sum_query_time) / deployment_sum_query_time * chargeable_pool\n ) * COALESCE(conf_chargeable_unit_rate, conf_ecu_rate)\n| STATS \n agg_querying = sum(querying)\n BY \n @timestamp,\n tier\n| WHERE agg_querying > 0" - }, - "visualization": { - "axisTitlesVisibilitySettings": { - "x": true, - "yLeft": true, - "yRight": true - }, - "fittingFunction": "Linear", - "gridlinesVisibilitySettings": { - "x": true, - "yLeft": true, - "yRight": true - }, - "labelsOrientation": { - "x": 0, - "yLeft": 0, - "yRight": 0 - }, - "layers": [ - { - "accessors": [ - "agg_querying" - ], - "colorMapping": { - "assignments": [], - "colorMode": { - "type": "categorical" - }, - "paletteId": "eui_amsterdam_color_blind", - "specialAssignments": [ - { - "color": { - "type": "loop" - }, - "rules": [ - { - "type": "other" - } - ], - "touched": false - } - ] - }, - "layerId": "61842c44-afd9-429c-bf2e-18a4a079e5e8", - "layerType": "data", - "seriesType": "bar_stacked", - "splitAccessor": "tier", - "splitAccessors": [ - "tier" - ], - "xAccessor": "@timestamp" - } - ], - "legend": { - "isVisible": false, - "position": "right", - "showSingleSeries": false - }, - "preferredSeriesType": "bar_stacked", - "tickLabelsVisibilitySettings": { - "x": true, - "yLeft": true, - "yRight": true - }, - "valueLabels": "hide" - } + "title": "Indexing Cost by Tier over Time", + "version": 2, + "visualizationType": "lnsXY" + }, + "title": "Indexing Cost by Tier over Time" + }, + "gridData": { + "h": 14, + "i": "2f961fd2-5b04-4c4d-aafb-76af69b06130", + "sectionId": "7f7cc9b5-13b3-4962-98f6-5b18a1d8ca80", + "w": 16, + "x": 16, + "y": 6 + }, + "panelIndex": "2f961fd2-5b04-4c4d-aafb-76af69b06130", + "type": "lens" + }, + { + "embeddableConfig": { + "attributes": { + "references": [], + "state": { + "datasourceStates": { + "textBased": { + "indexPatternRefs": [], + "layers": { + "d5520b0e-567f-4f0a-b424-fc79eb79e2cc": { + "columns": [ + { + "columnId": "@timestamp", + "customLabel": true, + "fieldName": "@timestamp", + "label": "Date", + "meta": { + "esType": "date", + "type": "date" + } }, - "title": "Query Cost by Tier over Time", - "version": 2, - "visualizationType": "lnsXY" - }, - "title": "Query Cost by Tier over Time" - }, - "gridData": { - "h": 14, - "i": "fb59beb6-bf05-4ce0-90a3-d8b434d300aa", - "sectionId": "7f7cc9b5-13b3-4962-98f6-5b18a1d8ca80", - "w": 16, - "x": 16, - "y": 23 - }, - "panelIndex": "fb59beb6-bf05-4ce0-90a3-d8b434d300aa", - "type": "lens" - }, - { - "embeddableConfig": { - "attributes": { - "references": [], - "state": { - "datasourceStates": { - "textBased": { - "indexPatternRefs": [], - "layers": { - "91bcc41e-3a2c-434d-8df7-29c01340e712": { - "columns": [ - { - "columnId": "agg_querying", - "customLabel": true, - "fieldName": "agg_querying", - "inMetricDimension": true, - "label": "% Normalized cost", - "meta": { - "esType": "double", - "type": "number" - }, - "params": { - "format": { - "id": "number", - "params": { - "decimals": 2 - } - } - } - }, - { - "columnId": "@timestamp", - "customLabel": true, - "fieldName": "@timestamp", - "label": "Date", - "meta": { - "esType": "date", - "type": "date" - } - }, - { - "columnId": "datastream", - "customLabel": true, - "fieldName": "datastream", - "label": "Data stream", - "meta": { - "esType": "keyword", - "type": "string" - } - } - ], - "index": "chargeback_integration", - "query": { - "esql": "FROM billing_realized_pool_lookup\n| LOOKUP JOIN cluster_capacity_utilization_lookup ON composite_key\n| LOOKUP JOIN cluster_deployment_contribution_lookup ON composite_key\n| LOOKUP JOIN cluster_datastream_contribution_lookup ON composite_key\n| LOOKUP JOIN chargeback_conf_lookup ON @timestamp >= conf_start_date AND @timestamp <= conf_end_date\n| EVAL heap_util = COALESCE(heap_used_pct_p95, 100) / 100, disk_util = COALESCE(disk_used_pct_p95, 100) / 100, w_sum = conf_utilization_memory_weight + conf_utilization_storage_weight, capacity_utilization_score = CASE(w_sum > 0, (conf_utilization_memory_weight * heap_util + conf_utilization_storage_weight * disk_util) / w_sum, 1.0)\n| EVAL chargeable_pool = data_tier_capacity_ecu * COALESCE(capacity_utilization_score, 1.0), utilization_data_missing = heap_used_pct_p95 IS NULL AND disk_used_pct_p95 IS NULL\n| EVAL \n querying = CASE (deployment_sum_query_time > 0, TO_DOUBLE(datastream_sum_query_time) / deployment_sum_query_time * chargeable_pool) * COALESCE(conf_chargeable_unit_rate, conf_ecu_rate)\n| STATS \n agg_querying = sum(querying)\n BY \n @timestamp, \n datastream\n| WHERE agg_querying > 0\n| SORT agg_querying DESC" - }, - "timeField": "@timestamp" - } - } - } - }, - "filters": [], - "needsRefresh": false, - "query": { - "esql": "FROM billing_realized_pool_lookup\n| LOOKUP JOIN cluster_capacity_utilization_lookup ON composite_key\n| LOOKUP JOIN cluster_deployment_contribution_lookup ON composite_key\n| LOOKUP JOIN cluster_datastream_contribution_lookup ON composite_key\n| LOOKUP JOIN chargeback_conf_lookup ON @timestamp >= conf_start_date AND @timestamp <= conf_end_date\n| EVAL heap_util = COALESCE(heap_used_pct_p95, 100) / 100, disk_util = COALESCE(disk_used_pct_p95, 100) / 100, w_sum = conf_utilization_memory_weight + conf_utilization_storage_weight, capacity_utilization_score = CASE(w_sum > 0, (conf_utilization_memory_weight * heap_util + conf_utilization_storage_weight * disk_util) / w_sum, 1.0)\n| EVAL chargeable_pool = data_tier_capacity_ecu * COALESCE(capacity_utilization_score, 1.0), utilization_data_missing = heap_used_pct_p95 IS NULL AND disk_used_pct_p95 IS NULL\n| EVAL \n querying = CASE (deployment_sum_query_time > 0, TO_DOUBLE(datastream_sum_query_time) / deployment_sum_query_time * chargeable_pool) * COALESCE(conf_chargeable_unit_rate, conf_ecu_rate)\n| STATS \n agg_querying = sum(querying)\n BY \n @timestamp, \n datastream\n| WHERE agg_querying > 0\n| SORT agg_querying DESC" - }, - "visualization": { - "axisTitlesVisibilitySettings": { - "x": true, - "yLeft": true, - "yRight": true - }, - "fittingFunction": "Linear", - "gridlinesVisibilitySettings": { - "x": true, - "yLeft": true, - "yRight": true - }, - "labelsOrientation": { - "x": 0, - "yLeft": 0, - "yRight": 0 - }, - "layers": [ - { - "accessors": [ - "agg_querying" - ], - "colorMapping": { - "assignments": [], - "colorMode": { - "type": "categorical" - }, - "paletteId": "eui_amsterdam_color_blind", - "specialAssignments": [ - { - "color": { - "type": "loop" - }, - "rules": [ - { - "type": "other" - } - ], - "touched": false - } - ] - }, - "layerId": "91bcc41e-3a2c-434d-8df7-29c01340e712", - "layerType": "data", - "seriesType": "bar_percentage_stacked", - "splitAccessor": "datastream", - "splitAccessors": [ - "datastream" - ], - "xAccessor": "@timestamp" - } - ], - "legend": { - "isVisible": false, - "position": "right", - "showSingleSeries": false - }, - "preferredSeriesType": "bar_stacked", - "tickLabelsVisibilitySettings": { - "x": true, - "yLeft": true, - "yRight": true - }, - "valueLabels": "hide" - } + { + "columnId": "datastream", + "customLabel": true, + "fieldName": "datastream", + "label": "Data stream", + "meta": { + "esType": "keyword", + "type": "string" + } }, - "title": "Query Cost Share by Data Stream over Time (%)", - "version": 2, - "visualizationType": "lnsXY" - }, - "title": "Query Cost Share by Data Stream over Time (%)" - }, - "gridData": { - "h": 14, - "i": "6fb65bcc-0568-4c8d-a833-c87cf8f8bccd", - "sectionId": "7f7cc9b5-13b3-4962-98f6-5b18a1d8ca80", - "w": 16, - "x": 32, - "y": 23 - }, - "panelIndex": "6fb65bcc-0568-4c8d-a833-c87cf8f8bccd", - "type": "lens" - }, - { - "embeddableConfig": { - "attributes": { - "references": [], - "state": { - "datasourceStates": { - "textBased": { - "indexPatternRefs": [], - "layers": { - "b72a0ab5-dbf5-4fd2-bdef-c855fe354bee": { - "columns": [ - { - "columnId": "agg_storage", - "customLabel": true, - "fieldName": "agg_storage", - "inMetricDimension": true, - "label": "Normalized cost", - "meta": { - "esType": "double", - "type": "number" - }, - "params": { - "format": { - "id": "number", - "params": { - "decimals": 2 - } - } - } - }, - { - "columnId": "@timestamp", - "customLabel": true, - "fieldName": "@timestamp", - "label": "Date", - "meta": { - "esType": "date", - "type": "date" - } - }, - { - "columnId": "datastream", - "customLabel": true, - "fieldName": "datastream", - "label": "Data stream", - "meta": { - "esType": "keyword", - "type": "string" - } - } - ], - "index": "chargeback_integration", - "query": { - "esql": "FROM billing_realized_pool_lookup\n| LOOKUP JOIN cluster_capacity_utilization_lookup ON composite_key\n| LOOKUP JOIN cluster_deployment_contribution_lookup ON composite_key\n| LOOKUP JOIN cluster_datastream_contribution_lookup ON composite_key\n| LOOKUP JOIN chargeback_conf_lookup ON @timestamp >= conf_start_date AND @timestamp <= conf_end_date\n| EVAL heap_util = COALESCE(heap_used_pct_p95, 100) / 100, disk_util = COALESCE(disk_used_pct_p95, 100) / 100, w_sum = conf_utilization_memory_weight + conf_utilization_storage_weight, capacity_utilization_score = CASE(w_sum > 0, (conf_utilization_memory_weight * heap_util + conf_utilization_storage_weight * disk_util) / w_sum, 1.0)\n| EVAL chargeable_pool = data_tier_capacity_ecu * COALESCE(capacity_utilization_score, 1.0), utilization_data_missing = heap_used_pct_p95 IS NULL AND disk_used_pct_p95 IS NULL\n| EVAL \n data_set = CASE (deployment_sum_data_set_store_size > 0, TO_DOUBLE(datastream_sum_data_set_store_size) / deployment_sum_data_set_store_size * chargeable_pool),\n store = CASE (deployment_sum_store_size > 0, TO_DOUBLE(datastream_sum_store_size) / deployment_sum_store_size * chargeable_pool),\n storage = CASE (store == 0, data_set, store) * COALESCE(conf_chargeable_unit_rate, conf_ecu_rate)\n| STATS \n agg_storage = sum(storage)\n BY \n @timestamp, \n datastream\n| WHERE agg_storage > 0\n| SORT agg_storage DESC" - }, - "timeField": "@timestamp" - } - } - } - }, - "filters": [], - "needsRefresh": false, - "query": { - "esql": "FROM billing_realized_pool_lookup\n| LOOKUP JOIN cluster_capacity_utilization_lookup ON composite_key\n| LOOKUP JOIN cluster_deployment_contribution_lookup ON composite_key\n| LOOKUP JOIN cluster_datastream_contribution_lookup ON composite_key\n| LOOKUP JOIN chargeback_conf_lookup ON @timestamp >= conf_start_date AND @timestamp <= conf_end_date\n| EVAL heap_util = COALESCE(heap_used_pct_p95, 100) / 100, disk_util = COALESCE(disk_used_pct_p95, 100) / 100, w_sum = conf_utilization_memory_weight + conf_utilization_storage_weight, capacity_utilization_score = CASE(w_sum > 0, (conf_utilization_memory_weight * heap_util + conf_utilization_storage_weight * disk_util) / w_sum, 1.0)\n| EVAL chargeable_pool = data_tier_capacity_ecu * COALESCE(capacity_utilization_score, 1.0), utilization_data_missing = heap_used_pct_p95 IS NULL AND disk_used_pct_p95 IS NULL\n| EVAL \n data_set = CASE (deployment_sum_data_set_store_size > 0, TO_DOUBLE(datastream_sum_data_set_store_size) / deployment_sum_data_set_store_size * chargeable_pool),\n store = CASE (deployment_sum_store_size > 0, TO_DOUBLE(datastream_sum_store_size) / deployment_sum_store_size * chargeable_pool),\n storage = CASE (store == 0, data_set, store) * COALESCE(conf_chargeable_unit_rate, conf_ecu_rate)\n| STATS \n agg_storage = sum(storage)\n BY \n @timestamp, \n datastream\n| WHERE agg_storage > 0\n| SORT agg_storage DESC" - }, - "visualization": { - "axisTitlesVisibilitySettings": { - "x": true, - "yLeft": true, - "yRight": true - }, - "fittingFunction": "Linear", - "gridlinesVisibilitySettings": { - "x": true, - "yLeft": true, - "yRight": true - }, - "labelsOrientation": { - "x": 0, - "yLeft": 0, - "yRight": 0 - }, - "layers": [ - { - "accessors": [ - "agg_storage" - ], - "colorMapping": { - "assignments": [], - "colorMode": { - "type": "categorical" - }, - "paletteId": "eui_amsterdam_color_blind", - "specialAssignments": [ - { - "color": { - "type": "loop" - }, - "rules": [ - { - "type": "other" - } - ], - "touched": false - } - ] - }, - "layerId": "b72a0ab5-dbf5-4fd2-bdef-c855fe354bee", - "layerType": "data", - "seriesType": "bar_stacked", - "splitAccessor": "datastream", - "splitAccessors": [ - "datastream" - ], - "xAccessor": "@timestamp" - } - ], - "legend": { - "isVisible": false, - "position": "right", - "showSingleSeries": false - }, - "preferredSeriesType": "bar_stacked", - "tickLabelsVisibilitySettings": { - "x": true, - "yLeft": true, - "yRight": true - }, - "valueLabels": "hide" + { + "columnId": "agg_indexing", + "customLabel": true, + "fieldName": "agg_indexing", + "inMetricDimension": true, + "label": "% Normalized cost", + "meta": { + "esType": "long", + "type": "number" + }, + "params": { + "format": { + "id": "percent", + "params": { + "decimals": 0 + } } - }, - "title": "Storage Cost by Data Stream over Time", - "version": 2, - "visualizationType": "lnsXY" - }, - "title": "Storage Cost by Data Stream over Time" - }, - "gridData": { - "h": 15, - "i": "5bdd9079-cb0e-453e-835b-afcb5d72f06b", - "sectionId": "7f7cc9b5-13b3-4962-98f6-5b18a1d8ca80", - "w": 16, - "x": 0, - "y": 40 - }, - "panelIndex": "5bdd9079-cb0e-453e-835b-afcb5d72f06b", - "type": "lens" - }, - { - "embeddableConfig": { - "attributes": { - "references": [], - "state": { - "datasourceStates": { - "textBased": { - "indexPatternRefs": [], - "layers": { - "93fd5f12-463f-4e8e-a9b8-0ac77d0522a9": { - "columns": [ - { - "columnId": "agg_storage", - "customLabel": true, - "fieldName": "agg_storage", - "inMetricDimension": true, - "label": "Normalized cost", - "meta": { - "esType": "double", - "type": "number" - }, - "params": { - "format": { - "id": "number", - "params": { - "decimals": 2 - } - } - } - }, - { - "columnId": "@timestamp", - "customLabel": true, - "fieldName": "@timestamp", - "label": "Date", - "meta": { - "esType": "date", - "type": "date" - } - }, - { - "columnId": "tier", - "customLabel": true, - "fieldName": "tier", - "label": "Data Tier", - "meta": { - "esType": "keyword", - "type": "string" - } - } - ], - "index": "chargeback_integration", - "query": { - "esql": "FROM billing_realized_pool_lookup\n| LOOKUP JOIN cluster_capacity_utilization_lookup ON composite_key\n| LOOKUP JOIN cluster_deployment_contribution_lookup ON composite_key\n| LOOKUP JOIN cluster_tier_contribution_lookup ON composite_key\n| LOOKUP JOIN chargeback_conf_lookup ON @timestamp >= conf_start_date AND @timestamp <= conf_end_date\n| EVAL heap_util = COALESCE(heap_used_pct_p95, 100) / 100, disk_util = COALESCE(disk_used_pct_p95, 100) / 100, w_sum = conf_utilization_memory_weight + conf_utilization_storage_weight, capacity_utilization_score = CASE(w_sum > 0, (conf_utilization_memory_weight * heap_util + conf_utilization_storage_weight * disk_util) / w_sum, 1.0)\n| EVAL chargeable_pool = data_tier_capacity_ecu * COALESCE(capacity_utilization_score, 1.0), utilization_data_missing = heap_used_pct_p95 IS NULL AND disk_used_pct_p95 IS NULL\n| EVAL \n data_set = CASE (deployment_sum_data_set_store_size > 0, TO_DOUBLE(tier_sum_data_set_store_size) / deployment_sum_data_set_store_size * chargeable_pool),\n store = CASE (deployment_sum_store_size > 0, TO_DOUBLE(tier_sum_store_size) / deployment_sum_store_size * chargeable_pool),\n storage = CASE (store == 0, data_set, store) * COALESCE(conf_chargeable_unit_rate, conf_ecu_rate)\n| STATS \n agg_storage = sum(storage)\n BY \n @timestamp,\n tier\n| WHERE agg_storage > 0" - }, - "timeField": "@timestamp" - } - } - } - }, - "filters": [], - "needsRefresh": false, - "query": { - "esql": "FROM billing_realized_pool_lookup\n| LOOKUP JOIN cluster_capacity_utilization_lookup ON composite_key\n| LOOKUP JOIN cluster_deployment_contribution_lookup ON composite_key\n| LOOKUP JOIN cluster_tier_contribution_lookup ON composite_key\n| LOOKUP JOIN chargeback_conf_lookup ON @timestamp >= conf_start_date AND @timestamp <= conf_end_date\n| EVAL heap_util = COALESCE(heap_used_pct_p95, 100) / 100, disk_util = COALESCE(disk_used_pct_p95, 100) / 100, w_sum = conf_utilization_memory_weight + conf_utilization_storage_weight, capacity_utilization_score = CASE(w_sum > 0, (conf_utilization_memory_weight * heap_util + conf_utilization_storage_weight * disk_util) / w_sum, 1.0)\n| EVAL chargeable_pool = data_tier_capacity_ecu * COALESCE(capacity_utilization_score, 1.0), utilization_data_missing = heap_used_pct_p95 IS NULL AND disk_used_pct_p95 IS NULL\n| EVAL \n data_set = CASE (deployment_sum_data_set_store_size > 0, TO_DOUBLE(tier_sum_data_set_store_size) / deployment_sum_data_set_store_size * chargeable_pool),\n store = CASE (deployment_sum_store_size > 0, TO_DOUBLE(tier_sum_store_size) / deployment_sum_store_size * chargeable_pool),\n storage = CASE (store == 0, data_set, store) * COALESCE(conf_chargeable_unit_rate, conf_ecu_rate)\n| STATS \n agg_storage = sum(storage)\n BY \n @timestamp,\n tier\n| WHERE agg_storage > 0" - }, - "visualization": { - "axisTitlesVisibilitySettings": { - "x": true, - "yLeft": true, - "yRight": true - }, - "fittingFunction": "Linear", - "gridlinesVisibilitySettings": { - "x": true, - "yLeft": true, - "yRight": true - }, - "labelsOrientation": { - "x": 0, - "yLeft": 0, - "yRight": 0 - }, - "layers": [ - { - "accessors": [ - "agg_storage" - ], - "colorMapping": { - "assignments": [], - "colorMode": { - "type": "categorical" - }, - "paletteId": "eui_amsterdam_color_blind", - "specialAssignments": [ - { - "color": { - "type": "loop" - }, - "rules": [ - { - "type": "other" - } - ], - "touched": false - } - ] - }, - "layerId": "93fd5f12-463f-4e8e-a9b8-0ac77d0522a9", - "layerType": "data", - "seriesType": "bar_stacked", - "splitAccessor": "tier", - "splitAccessors": [ - "tier" - ], - "xAccessor": "@timestamp" - } - ], - "legend": { - "isVisible": false, - "position": "right", - "showSingleSeries": false - }, - "preferredSeriesType": "bar_stacked", - "tickLabelsVisibilitySettings": { - "x": true, - "yLeft": true, - "yRight": true - }, - "valueLabels": "hide" + } + } + ], + "index": "chargeback_integration", + "query": { + "esql": "FROM billing_realized_pool_lookup\n| LOOKUP JOIN cluster_capacity_utilization_lookup ON composite_key\n| LOOKUP JOIN cluster_deployment_contribution_lookup ON composite_key\n| LOOKUP JOIN cluster_datastream_contribution_lookup ON composite_key\n| LOOKUP JOIN chargeback_conf_lookup ON @timestamp >= conf_start_date AND @timestamp <= conf_end_date\n| EVAL heap_util = COALESCE(heap_used_pct_p95, 100) / 100, disk_util = COALESCE(disk_used_pct_p95, 100) / 100, w_sum = conf_utilization_memory_weight + conf_utilization_storage_weight, capacity_utilization_score = CASE(w_sum > 0, (conf_utilization_memory_weight * heap_util + conf_utilization_storage_weight * disk_util) / w_sum, 1.0)\n| EVAL chargeable_pool = data_tier_capacity_ecu * COALESCE(capacity_utilization_score, 1.0), utilization_data_missing = heap_used_pct_p95 IS NULL AND disk_used_pct_p95 IS NULL\n| EVAL \n indexing = CASE (\n deployment_sum_indexing_time > 0,\n TO_DOUBLE(datastream_sum_indexing_time) / deployment_sum_indexing_time * chargeable_pool\n ) * COALESCE(conf_chargeable_unit_rate, conf_ecu_rate)\n| STATS \n agg_indexing = sum(indexing)\n BY \n @timestamp,\n datastream\n| WHERE agg_indexing > 0\n| SORT agg_indexing DESC" + }, + "timeField": "@timestamp" + } + } + } + }, + "filters": [], + "needsRefresh": false, + "query": { + "esql": "FROM billing_realized_pool_lookup\n| LOOKUP JOIN cluster_capacity_utilization_lookup ON composite_key\n| LOOKUP JOIN cluster_deployment_contribution_lookup ON composite_key\n| LOOKUP JOIN cluster_datastream_contribution_lookup ON composite_key\n| LOOKUP JOIN chargeback_conf_lookup ON @timestamp >= conf_start_date AND @timestamp <= conf_end_date\n| EVAL heap_util = COALESCE(heap_used_pct_p95, 100) / 100, disk_util = COALESCE(disk_used_pct_p95, 100) / 100, w_sum = conf_utilization_memory_weight + conf_utilization_storage_weight, capacity_utilization_score = CASE(w_sum > 0, (conf_utilization_memory_weight * heap_util + conf_utilization_storage_weight * disk_util) / w_sum, 1.0)\n| EVAL chargeable_pool = data_tier_capacity_ecu * COALESCE(capacity_utilization_score, 1.0), utilization_data_missing = heap_used_pct_p95 IS NULL AND disk_used_pct_p95 IS NULL\n| EVAL \n indexing = CASE (\n deployment_sum_indexing_time > 0,\n TO_DOUBLE(datastream_sum_indexing_time) / deployment_sum_indexing_time * chargeable_pool\n ) * COALESCE(conf_chargeable_unit_rate, conf_ecu_rate)\n| STATS \n agg_indexing = sum(indexing)\n BY \n @timestamp,\n datastream\n| WHERE agg_indexing > 0\n| SORT agg_indexing DESC" + }, + "visualization": { + "axisTitlesVisibilitySettings": { + "x": true, + "yLeft": true, + "yRight": true + }, + "fittingFunction": "Linear", + "gridlinesVisibilitySettings": { + "x": true, + "yLeft": true, + "yRight": true + }, + "labelsOrientation": { + "x": 0, + "yLeft": 0, + "yRight": 0 + }, + "layers": [ + { + "accessors": [ + "agg_indexing" + ], + "colorMapping": { + "assignments": [], + "colorMode": { + "type": "categorical" + }, + "paletteId": "eui_amsterdam_color_blind", + "specialAssignments": [ + { + "color": { + "type": "loop" + }, + "rules": [ + { + "type": "other" } - }, - "title": "Storage Cost by Tier over Time", - "version": 2, - "visualizationType": "lnsXY" + ], + "touched": false + } + ] }, - "title": "Storage Cost by Tier over Time" - }, - "gridData": { - "h": 15, - "i": "5ac2dcd4-8775-4c15-8e79-346e0a62dd4d", - "sectionId": "7f7cc9b5-13b3-4962-98f6-5b18a1d8ca80", - "w": 16, - "x": 16, - "y": 40 - }, - "panelIndex": "5ac2dcd4-8775-4c15-8e79-346e0a62dd4d", - "type": "lens" + "layerId": "d5520b0e-567f-4f0a-b424-fc79eb79e2cc", + "layerType": "data", + "seriesType": "bar_percentage_stacked", + "splitAccessor": "datastream", + "splitAccessors": [ + "datastream" + ], + "xAccessor": "@timestamp" + } + ], + "legend": { + "isVisible": false, + "position": "right", + "showSingleSeries": false + }, + "preferredSeriesType": "bar_stacked", + "tickLabelsVisibilitySettings": { + "x": true, + "yLeft": true, + "yRight": true + }, + "valueLabels": "hide" + } }, - { - "embeddableConfig": { - "attributes": { - "references": [], - "state": { - "datasourceStates": { - "textBased": { - "indexPatternRefs": [], - "layers": { - "b72a0ab5-dbf5-4fd2-bdef-c855fe354bee": { - "columns": [ - { - "columnId": "agg_storage", - "customLabel": true, - "fieldName": "agg_storage", - "inMetricDimension": true, - "label": "% Normalized cost", - "meta": { - "esType": "double", - "type": "number" - }, - "params": { - "format": { - "id": "number", - "params": { - "decimals": 2 - } - } - } - }, - { - "columnId": "@timestamp", - "customLabel": true, - "fieldName": "@timestamp", - "label": "Date", - "meta": { - "esType": "date", - "type": "date" - } - }, - { - "columnId": "datastream", - "customLabel": true, - "fieldName": "datastream", - "label": "Data stream", - "meta": { - "esType": "keyword", - "type": "string" - } - } - ], - "index": "chargeback_integration", - "query": { - "esql": "FROM billing_realized_pool_lookup\n| LOOKUP JOIN cluster_capacity_utilization_lookup ON composite_key\n| LOOKUP JOIN cluster_deployment_contribution_lookup ON composite_key\n| LOOKUP JOIN cluster_datastream_contribution_lookup ON composite_key\n| LOOKUP JOIN chargeback_conf_lookup ON @timestamp >= conf_start_date AND @timestamp <= conf_end_date\n| EVAL heap_util = COALESCE(heap_used_pct_p95, 100) / 100, disk_util = COALESCE(disk_used_pct_p95, 100) / 100, w_sum = conf_utilization_memory_weight + conf_utilization_storage_weight, capacity_utilization_score = CASE(w_sum > 0, (conf_utilization_memory_weight * heap_util + conf_utilization_storage_weight * disk_util) / w_sum, 1.0)\n| EVAL chargeable_pool = data_tier_capacity_ecu * COALESCE(capacity_utilization_score, 1.0), utilization_data_missing = heap_used_pct_p95 IS NULL AND disk_used_pct_p95 IS NULL\n| EVAL \n data_set = CASE (deployment_sum_data_set_store_size > 0, TO_DOUBLE(datastream_sum_data_set_store_size) / deployment_sum_data_set_store_size * chargeable_pool),\n store = CASE (deployment_sum_store_size > 0, TO_DOUBLE(datastream_sum_store_size) / deployment_sum_store_size * chargeable_pool),\n storage = CASE (store == 0, data_set, store) * COALESCE(conf_chargeable_unit_rate, conf_ecu_rate)\n| STATS \n agg_storage = sum(storage)\n BY \n @timestamp, \n datastream\n| WHERE agg_storage > 0\n| SORT agg_storage DESC" - }, - "timeField": "@timestamp" - } - } - } - }, - "filters": [], - "needsRefresh": false, - "query": { - "esql": "FROM billing_realized_pool_lookup\n| LOOKUP JOIN cluster_capacity_utilization_lookup ON composite_key\n| LOOKUP JOIN cluster_deployment_contribution_lookup ON composite_key\n| LOOKUP JOIN cluster_datastream_contribution_lookup ON composite_key\n| LOOKUP JOIN chargeback_conf_lookup ON @timestamp >= conf_start_date AND @timestamp <= conf_end_date\n| EVAL heap_util = COALESCE(heap_used_pct_p95, 100) / 100, disk_util = COALESCE(disk_used_pct_p95, 100) / 100, w_sum = conf_utilization_memory_weight + conf_utilization_storage_weight, capacity_utilization_score = CASE(w_sum > 0, (conf_utilization_memory_weight * heap_util + conf_utilization_storage_weight * disk_util) / w_sum, 1.0)\n| EVAL chargeable_pool = data_tier_capacity_ecu * COALESCE(capacity_utilization_score, 1.0), utilization_data_missing = heap_used_pct_p95 IS NULL AND disk_used_pct_p95 IS NULL\n| EVAL \n data_set = CASE (deployment_sum_data_set_store_size > 0, TO_DOUBLE(datastream_sum_data_set_store_size) / deployment_sum_data_set_store_size * chargeable_pool),\n store = CASE (deployment_sum_store_size > 0, TO_DOUBLE(datastream_sum_store_size) / deployment_sum_store_size * chargeable_pool),\n storage = CASE (store == 0, data_set, store) * COALESCE(conf_chargeable_unit_rate, conf_ecu_rate)\n| STATS \n agg_storage = sum(storage)\n BY \n @timestamp, \n datastream\n| WHERE agg_storage > 0\n| SORT agg_storage DESC" - }, - "visualization": { - "axisTitlesVisibilitySettings": { - "x": true, - "yLeft": true, - "yRight": true - }, - "fittingFunction": "Linear", - "gridlinesVisibilitySettings": { - "x": true, - "yLeft": true, - "yRight": true - }, - "labelsOrientation": { - "x": 0, - "yLeft": 0, - "yRight": 0 - }, - "layers": [ - { - "accessors": [ - "agg_storage" - ], - "colorMapping": { - "assignments": [], - "colorMode": { - "type": "categorical" - }, - "paletteId": "eui_amsterdam_color_blind", - "specialAssignments": [ - { - "color": { - "type": "loop" - }, - "rules": [ - { - "type": "other" - } - ], - "touched": false - } - ] - }, - "layerId": "b72a0ab5-dbf5-4fd2-bdef-c855fe354bee", - "layerType": "data", - "seriesType": "bar_percentage_stacked", - "splitAccessor": "datastream", - "splitAccessors": [ - "datastream" - ], - "xAccessor": "@timestamp" - } - ], - "legend": { - "isVisible": false, - "position": "right", - "showSingleSeries": false - }, - "preferredSeriesType": "bar_stacked", - "tickLabelsVisibilitySettings": { - "x": true, - "yLeft": true, - "yRight": true - }, - "valueLabels": "hide" + "title": "Indexing Cost Share by Data Stream over Time (%)", + "version": 2, + "visualizationType": "lnsXY" + }, + "title": "Indexing Cost Share by Data Stream over Time (%)" + }, + "gridData": { + "h": 14, + "i": "c895d51f-09ee-4624-8674-1d77a8d1df61", + "sectionId": "7f7cc9b5-13b3-4962-98f6-5b18a1d8ca80", + "w": 16, + "x": 32, + "y": 6 + }, + "panelIndex": "c895d51f-09ee-4624-8674-1d77a8d1df61", + "type": "lens" + }, + { + "embeddableConfig": { + "attributes": { + "references": [], + "state": { + "datasourceStates": { + "textBased": { + "indexPatternRefs": [], + "layers": { + "1a2f8269-f56f-4c02-8506-76b723acc828": { + "columns": [ + { + "columnId": "agg_querying", + "customLabel": true, + "fieldName": "agg_querying", + "inMetricDimension": true, + "label": "Normalized cost", + "meta": { + "esType": "double", + "type": "number" + }, + "params": { + "format": { + "id": "number", + "params": { + "decimals": 2 + } } + } }, - "title": "Storage Cost Share by Data Stream over Time (%)", - "version": 2, - "visualizationType": "lnsXY" + { + "columnId": "@timestamp", + "customLabel": true, + "fieldName": "@timestamp", + "label": "Date", + "meta": { + "esType": "date", + "type": "date" + } + }, + { + "columnId": "datastream", + "customLabel": true, + "fieldName": "datastream", + "label": "Data stream", + "meta": { + "esType": "keyword", + "type": "string" + } + } + ], + "index": "chargeback_integration", + "query": { + "esql": "FROM billing_realized_pool_lookup\n| LOOKUP JOIN cluster_capacity_utilization_lookup ON composite_key\n| LOOKUP JOIN cluster_deployment_contribution_lookup ON composite_key\n| LOOKUP JOIN cluster_datastream_contribution_lookup ON composite_key\n| LOOKUP JOIN chargeback_conf_lookup ON @timestamp >= conf_start_date AND @timestamp <= conf_end_date\n| EVAL heap_util = COALESCE(heap_used_pct_p95, 100) / 100, disk_util = COALESCE(disk_used_pct_p95, 100) / 100, w_sum = conf_utilization_memory_weight + conf_utilization_storage_weight, capacity_utilization_score = CASE(w_sum > 0, (conf_utilization_memory_weight * heap_util + conf_utilization_storage_weight * disk_util) / w_sum, 1.0)\n| EVAL chargeable_pool = data_tier_capacity_ecu * COALESCE(capacity_utilization_score, 1.0), utilization_data_missing = heap_used_pct_p95 IS NULL AND disk_used_pct_p95 IS NULL\n| EVAL \n querying = CASE (\n deployment_sum_query_time > 0,\n TO_DOUBLE(datastream_sum_query_time) / deployment_sum_query_time * chargeable_pool\n ) * COALESCE(conf_chargeable_unit_rate, conf_ecu_rate)\n| STATS \n agg_querying = sum(querying) \n BY \n @timestamp, \n datastream\n| WHERE agg_querying > 0\n| SORT agg_querying DESC" + }, + "timeField": "@timestamp" + } + } + } + }, + "filters": [], + "needsRefresh": false, + "query": { + "esql": "FROM billing_realized_pool_lookup\n| LOOKUP JOIN cluster_capacity_utilization_lookup ON composite_key\n| LOOKUP JOIN cluster_deployment_contribution_lookup ON composite_key\n| LOOKUP JOIN cluster_datastream_contribution_lookup ON composite_key\n| LOOKUP JOIN chargeback_conf_lookup ON @timestamp >= conf_start_date AND @timestamp <= conf_end_date\n| EVAL heap_util = COALESCE(heap_used_pct_p95, 100) / 100, disk_util = COALESCE(disk_used_pct_p95, 100) / 100, w_sum = conf_utilization_memory_weight + conf_utilization_storage_weight, capacity_utilization_score = CASE(w_sum > 0, (conf_utilization_memory_weight * heap_util + conf_utilization_storage_weight * disk_util) / w_sum, 1.0)\n| EVAL chargeable_pool = data_tier_capacity_ecu * COALESCE(capacity_utilization_score, 1.0), utilization_data_missing = heap_used_pct_p95 IS NULL AND disk_used_pct_p95 IS NULL\n| EVAL \n querying = CASE (\n deployment_sum_query_time > 0,\n TO_DOUBLE(datastream_sum_query_time) / deployment_sum_query_time * chargeable_pool\n ) * COALESCE(conf_chargeable_unit_rate, conf_ecu_rate)\n| STATS \n agg_querying = sum(querying) \n BY \n @timestamp, \n datastream\n| WHERE agg_querying > 0\n| SORT agg_querying DESC" + }, + "visualization": { + "axisTitlesVisibilitySettings": { + "x": true, + "yLeft": true, + "yRight": true + }, + "fittingFunction": "Linear", + "gridlinesVisibilitySettings": { + "x": true, + "yLeft": true, + "yRight": true + }, + "labelsOrientation": { + "x": 0, + "yLeft": 0, + "yRight": 0 + }, + "layers": [ + { + "accessors": [ + "agg_querying" + ], + "colorMapping": { + "assignments": [], + "colorMode": { + "type": "categorical" + }, + "paletteId": "eui_amsterdam_color_blind", + "specialAssignments": [ + { + "color": { + "type": "loop" + }, + "rules": [ + { + "type": "other" + } + ], + "touched": false + } + ] }, - "title": "Storage Cost Share by Data Stream over Time (%)" - }, - "gridData": { - "h": 15, - "i": "5d96941f-c4e3-4833-b716-192d75d6ae0d", - "sectionId": "7f7cc9b5-13b3-4962-98f6-5b18a1d8ca80", - "w": 16, - "x": 32, - "y": 40 - }, - "panelIndex": "5d96941f-c4e3-4833-b716-192d75d6ae0d", - "type": "lens" + "layerId": "1a2f8269-f56f-4c02-8506-76b723acc828", + "layerType": "data", + "seriesType": "bar_stacked", + "splitAccessor": "datastream", + "splitAccessors": [ + "datastream" + ], + "xAccessor": "@timestamp" + } + ], + "legend": { + "isVisible": false, + "position": "right", + "showSingleSeries": false + }, + "preferredSeriesType": "bar_stacked", + "tickLabelsVisibilitySettings": { + "x": true, + "yLeft": true, + "yRight": true + }, + "valueLabels": "hide" + } }, - { - "embeddableConfig": { - "attributes": { - "references": [], - "state": { - "datasourceStates": { - "textBased": { - "indexPatternRefs": [], - "layers": { - "52906f25-a114-4d45-b425-f0ff154f4ac1": { - "columns": [ - { - "columnId": "agg_blended", - "customLabel": true, - "fieldName": "agg_blended", - "inMetricDimension": true, - "label": "Normalized cost", - "meta": { - "esType": "double", - "type": "number" - }, - "params": { - "format": { - "id": "number", - "params": { - "decimals": 2 - } - } - } - }, - { - "columnId": "@timestamp", - "customLabel": true, - "fieldName": "@timestamp", - "label": "Date", - "meta": { - "esType": "date", - "type": "date" - } - }, - { - "columnId": "datastream", - "customLabel": false, - "fieldName": "datastream", - "label": "datastream", - "meta": { - "esType": "keyword", - "type": "string" - } - } - ], - "index": "chargeback_integration", - "query": { - "esql": "FROM billing_realized_pool_lookup\n| LOOKUP JOIN cluster_capacity_utilization_lookup ON composite_key\n| LOOKUP JOIN cluster_deployment_contribution_lookup ON composite_key\n| LOOKUP JOIN cluster_datastream_contribution_lookup ON composite_key\n| LOOKUP JOIN chargeback_conf_lookup ON @timestamp >= conf_start_date AND @timestamp <= conf_end_date\n| EVAL heap_util = COALESCE(heap_used_pct_p95, 100) / 100, disk_util = COALESCE(disk_used_pct_p95, 100) / 100, w_sum = conf_utilization_memory_weight + conf_utilization_storage_weight, capacity_utilization_score = CASE(w_sum > 0, (conf_utilization_memory_weight * heap_util + conf_utilization_storage_weight * disk_util) / w_sum, 1.0)\n| EVAL chargeable_pool = data_tier_capacity_ecu * COALESCE(capacity_utilization_score, 1.0), utilization_data_missing = heap_used_pct_p95 IS NULL AND disk_used_pct_p95 IS NULL\n| EVAL\n indexing = CASE (deployment_sum_indexing_time > 0, TO_DOUBLE(datastream_sum_indexing_time) / deployment_sum_indexing_time * chargeable_pool),\n querying = CASE (deployment_sum_query_time > 0, TO_DOUBLE(datastream_sum_query_time) / deployment_sum_query_time * chargeable_pool),\n data_set = CASE (deployment_sum_data_set_store_size > 0, TO_DOUBLE(datastream_sum_data_set_store_size) / deployment_sum_data_set_store_size * chargeable_pool),\n store = CASE (deployment_sum_store_size > 0, TO_DOUBLE(datastream_sum_store_size) / deployment_sum_store_size * chargeable_pool),\n storage = CASE (store == 0, data_set, store),\n total_weight_hot = conf_storage_weight + conf_query_weight + conf_indexing_weight,\n blended = TO_DOUBLE(((storage * conf_storage_weight) + (querying * conf_query_weight) + (indexing * conf_indexing_weight))) / total_weight_hot * COALESCE(conf_chargeable_unit_rate, conf_ecu_rate)\n| STATS\n agg_blended = sum(blended)\n BY\n @timestamp,\n datastream\n| WHERE agg_blended > 0\n| SORT agg_blended DESC" - }, - "timeField": "@timestamp" - } - } - } - }, - "filters": [], - "needsRefresh": false, - "query": { - "esql": "FROM billing_realized_pool_lookup\n| LOOKUP JOIN cluster_capacity_utilization_lookup ON composite_key\n| LOOKUP JOIN cluster_deployment_contribution_lookup ON composite_key\n| LOOKUP JOIN cluster_datastream_contribution_lookup ON composite_key\n| LOOKUP JOIN chargeback_conf_lookup ON @timestamp >= conf_start_date AND @timestamp <= conf_end_date\n| EVAL heap_util = COALESCE(heap_used_pct_p95, 100) / 100, disk_util = COALESCE(disk_used_pct_p95, 100) / 100, w_sum = conf_utilization_memory_weight + conf_utilization_storage_weight, capacity_utilization_score = CASE(w_sum > 0, (conf_utilization_memory_weight * heap_util + conf_utilization_storage_weight * disk_util) / w_sum, 1.0)\n| EVAL chargeable_pool = data_tier_capacity_ecu * COALESCE(capacity_utilization_score, 1.0), utilization_data_missing = heap_used_pct_p95 IS NULL AND disk_used_pct_p95 IS NULL\n| EVAL\n indexing = CASE (deployment_sum_indexing_time > 0, TO_DOUBLE(datastream_sum_indexing_time) / deployment_sum_indexing_time * chargeable_pool),\n querying = CASE (deployment_sum_query_time > 0, TO_DOUBLE(datastream_sum_query_time) / deployment_sum_query_time * chargeable_pool),\n data_set = CASE (deployment_sum_data_set_store_size > 0, TO_DOUBLE(datastream_sum_data_set_store_size) / deployment_sum_data_set_store_size * chargeable_pool),\n store = CASE (deployment_sum_store_size > 0, TO_DOUBLE(datastream_sum_store_size) / deployment_sum_store_size * chargeable_pool),\n storage = CASE (store == 0, data_set, store),\n total_weight_hot = conf_storage_weight + conf_query_weight + conf_indexing_weight,\n blended = TO_DOUBLE(((storage * conf_storage_weight) + (querying * conf_query_weight) + (indexing * conf_indexing_weight))) / total_weight_hot * COALESCE(conf_chargeable_unit_rate, conf_ecu_rate)\n| STATS\n agg_blended = sum(blended)\n BY\n @timestamp,\n datastream\n| WHERE agg_blended > 0\n| SORT agg_blended DESC" - }, - "visualization": { - "axisTitlesVisibilitySettings": { - "x": true, - "yLeft": true, - "yRight": true - }, - "fittingFunction": "Linear", - "gridlinesVisibilitySettings": { - "x": true, - "yLeft": true, - "yRight": true - }, - "labelsOrientation": { - "x": 0, - "yLeft": 0, - "yRight": 0 - }, - "layers": [ - { - "accessors": [ - "agg_blended" - ], - "colorMapping": { - "assignments": [], - "colorMode": { - "type": "categorical" - }, - "paletteId": "eui_amsterdam_color_blind", - "specialAssignments": [ - { - "color": { - "type": "loop" - }, - "rules": [ - { - "type": "other" - } - ], - "touched": false - } - ] - }, - "layerId": "52906f25-a114-4d45-b425-f0ff154f4ac1", - "layerType": "data", - "seriesType": "bar_stacked", - "splitAccessor": "datastream", - "splitAccessors": [ - "datastream" - ], - "xAccessor": "@timestamp" - } - ], - "legend": { - "isVisible": false, - "position": "right", - "showSingleSeries": false - }, - "preferredSeriesType": "bar_stacked", - "tickLabelsVisibilitySettings": { - "x": true, - "yLeft": true, - "yRight": true - }, - "valueLabels": "hide" + "title": "Query Cost by Data Stream over Time", + "version": 2, + "visualizationType": "lnsXY" + }, + "title": "Query Cost by Data Stream over Time" + }, + "gridData": { + "h": 14, + "i": "48aef73e-b6de-424a-85fa-74f46d9bded4", + "sectionId": "7f7cc9b5-13b3-4962-98f6-5b18a1d8ca80", + "w": 16, + "x": 0, + "y": 23 + }, + "panelIndex": "48aef73e-b6de-424a-85fa-74f46d9bded4", + "type": "lens" + }, + { + "embeddableConfig": { + "attributes": { + "references": [], + "state": { + "datasourceStates": { + "textBased": { + "indexPatternRefs": [], + "layers": { + "61842c44-afd9-429c-bf2e-18a4a079e5e8": { + "columns": [ + { + "columnId": "agg_querying", + "customLabel": true, + "fieldName": "agg_querying", + "inMetricDimension": true, + "label": "Normalized cost", + "meta": { + "esType": "double", + "type": "number" + }, + "params": { + "format": { + "id": "number", + "params": { + "decimals": 2 + } } + } + }, + { + "columnId": "@timestamp", + "customLabel": true, + "fieldName": "@timestamp", + "label": "Date", + "meta": { + "esType": "date", + "type": "date" + } }, - "title": "Blended Cost by Data Stream over Time", - "version": 2, - "visualizationType": "lnsXY" + { + "columnId": "tier", + "customLabel": true, + "fieldName": "tier", + "label": "Data Tier", + "meta": { + "esType": "keyword", + "type": "string" + } + } + ], + "index": "chargeback_integration", + "query": { + "esql": "FROM billing_realized_pool_lookup\n| LOOKUP JOIN cluster_capacity_utilization_lookup ON composite_key\n| LOOKUP JOIN cluster_deployment_contribution_lookup ON composite_key\n| LOOKUP JOIN cluster_tier_contribution_lookup ON composite_key\n| LOOKUP JOIN chargeback_conf_lookup ON @timestamp >= conf_start_date AND @timestamp <= conf_end_date\n| EVAL heap_util = COALESCE(heap_used_pct_p95, 100) / 100, disk_util = COALESCE(disk_used_pct_p95, 100) / 100, w_sum = conf_utilization_memory_weight + conf_utilization_storage_weight, capacity_utilization_score = CASE(w_sum > 0, (conf_utilization_memory_weight * heap_util + conf_utilization_storage_weight * disk_util) / w_sum, 1.0)\n| EVAL chargeable_pool = data_tier_capacity_ecu * COALESCE(capacity_utilization_score, 1.0), utilization_data_missing = heap_used_pct_p95 IS NULL AND disk_used_pct_p95 IS NULL\n| EVAL \n querying = CASE (\n deployment_sum_query_time > 0,\n TO_DOUBLE(tier_sum_query_time) / deployment_sum_query_time * chargeable_pool\n ) * COALESCE(conf_chargeable_unit_rate, conf_ecu_rate)\n| STATS \n agg_querying = sum(querying)\n BY \n @timestamp,\n tier\n| WHERE agg_querying > 0" + }, + "timeField": "@timestamp" + } + } + } + }, + "filters": [], + "needsRefresh": false, + "query": { + "esql": "FROM billing_realized_pool_lookup\n| LOOKUP JOIN cluster_capacity_utilization_lookup ON composite_key\n| LOOKUP JOIN cluster_deployment_contribution_lookup ON composite_key\n| LOOKUP JOIN cluster_tier_contribution_lookup ON composite_key\n| LOOKUP JOIN chargeback_conf_lookup ON @timestamp >= conf_start_date AND @timestamp <= conf_end_date\n| EVAL heap_util = COALESCE(heap_used_pct_p95, 100) / 100, disk_util = COALESCE(disk_used_pct_p95, 100) / 100, w_sum = conf_utilization_memory_weight + conf_utilization_storage_weight, capacity_utilization_score = CASE(w_sum > 0, (conf_utilization_memory_weight * heap_util + conf_utilization_storage_weight * disk_util) / w_sum, 1.0)\n| EVAL chargeable_pool = data_tier_capacity_ecu * COALESCE(capacity_utilization_score, 1.0), utilization_data_missing = heap_used_pct_p95 IS NULL AND disk_used_pct_p95 IS NULL\n| EVAL \n querying = CASE (\n deployment_sum_query_time > 0,\n TO_DOUBLE(tier_sum_query_time) / deployment_sum_query_time * chargeable_pool\n ) * COALESCE(conf_chargeable_unit_rate, conf_ecu_rate)\n| STATS \n agg_querying = sum(querying)\n BY \n @timestamp,\n tier\n| WHERE agg_querying > 0" + }, + "visualization": { + "axisTitlesVisibilitySettings": { + "x": true, + "yLeft": true, + "yRight": true + }, + "fittingFunction": "Linear", + "gridlinesVisibilitySettings": { + "x": true, + "yLeft": true, + "yRight": true + }, + "labelsOrientation": { + "x": 0, + "yLeft": 0, + "yRight": 0 + }, + "layers": [ + { + "accessors": [ + "agg_querying" + ], + "colorMapping": { + "assignments": [], + "colorMode": { + "type": "categorical" + }, + "paletteId": "eui_amsterdam_color_blind", + "specialAssignments": [ + { + "color": { + "type": "loop" + }, + "rules": [ + { + "type": "other" + } + ], + "touched": false + } + ] }, - "title": "Blended Cost by Data Stream over Time" - }, - "gridData": { - "h": 14, - "i": "0f25691b-8bc9-4dd2-94be-893a6e64a195", - "sectionId": "7f7cc9b5-13b3-4962-98f6-5b18a1d8ca80", - "w": 16, - "x": 0, - "y": 58 - }, - "panelIndex": "0f25691b-8bc9-4dd2-94be-893a6e64a195", - "type": "lens" + "layerId": "61842c44-afd9-429c-bf2e-18a4a079e5e8", + "layerType": "data", + "seriesType": "bar_stacked", + "splitAccessor": "tier", + "splitAccessors": [ + "tier" + ], + "xAccessor": "@timestamp" + } + ], + "legend": { + "isVisible": false, + "position": "right", + "showSingleSeries": false + }, + "preferredSeriesType": "bar_stacked", + "tickLabelsVisibilitySettings": { + "x": true, + "yLeft": true, + "yRight": true + }, + "valueLabels": "hide" + } }, - { - "embeddableConfig": { - "attributes": { - "references": [], - "state": { - "datasourceStates": { - "textBased": { - "indexPatternRefs": [], - "layers": { - "43f9cfe6-d828-4773-ab65-129f92cf7f4d": { - "columns": [ - { - "columnId": "@timestamp", - "customLabel": false, - "fieldName": "@timestamp", - "label": "@timestamp", - "meta": { - "esType": "date", - "type": "date" - } - }, - { - "columnId": "Data Tier", - "customLabel": false, - "fieldName": "Data Tier", - "label": "Data Tier", - "meta": { - "esType": "keyword", - "type": "string" - } - }, - { - "columnId": "Normalized cost", - "customLabel": false, - "fieldName": "Normalized cost", - "inMetricDimension": true, - "label": "Normalized cost", - "meta": { - "esType": "double", - "type": "number" - } - } - ], - "index": "chargeback_integration", - "query": { - "esql": "FROM billing_realized_pool_lookup\n| LOOKUP JOIN cluster_capacity_utilization_lookup ON composite_key\n| LOOKUP JOIN cluster_deployment_contribution_lookup ON composite_key\n| LOOKUP JOIN cluster_tier_contribution_lookup ON composite_key\n| LOOKUP JOIN chargeback_conf_lookup ON @timestamp >= conf_start_date AND @timestamp <= conf_end_date\n| EVAL heap_util = COALESCE(heap_used_pct_p95, 100) / 100, disk_util = COALESCE(disk_used_pct_p95, 100) / 100, w_sum = conf_utilization_memory_weight + conf_utilization_storage_weight, capacity_utilization_score = CASE(w_sum > 0, (conf_utilization_memory_weight * heap_util + conf_utilization_storage_weight * disk_util) / w_sum, 1.0)\n| EVAL chargeable_pool = data_tier_capacity_ecu * COALESCE(capacity_utilization_score, 1.0), utilization_data_missing = heap_used_pct_p95 IS NULL AND disk_used_pct_p95 IS NULL\n| EVAL\n indexing = CASE (deployment_sum_indexing_time > 0, TO_DOUBLE(tier_sum_indexing_time) / deployment_sum_indexing_time * chargeable_pool) * COALESCE(conf_chargeable_unit_rate, conf_ecu_rate),\n querying = CASE (deployment_sum_query_time > 0, TO_DOUBLE(tier_sum_query_time) / deployment_sum_query_time * chargeable_pool) * COALESCE(conf_chargeable_unit_rate, conf_ecu_rate),\n data_set = CASE (deployment_sum_data_set_store_size > 0, TO_DOUBLE(tier_sum_data_set_store_size) / deployment_sum_data_set_store_size * chargeable_pool),\n store = CASE (deployment_sum_store_size > 0, TO_DOUBLE(tier_sum_store_size) / deployment_sum_store_size * chargeable_pool),\n storage = CASE (store == 0, data_set, store) * COALESCE(conf_chargeable_unit_rate, conf_ecu_rate),\n total_weight_hot = conf_storage_weight + conf_query_weight + conf_indexing_weight,\n total_weight_cold = conf_storage_weight + conf_query_weight,\n blended = CASE (\n tier == \"hot/content\",\n ((storage / COALESCE(conf_chargeable_unit_rate, conf_ecu_rate) * conf_storage_weight) + (querying / COALESCE(conf_chargeable_unit_rate, conf_ecu_rate) * conf_query_weight) + (indexing / COALESCE(conf_chargeable_unit_rate, conf_ecu_rate) * conf_indexing_weight)) / total_weight_hot,\n ((storage / COALESCE(conf_chargeable_unit_rate, conf_ecu_rate) * conf_storage_weight) + (querying / COALESCE(conf_chargeable_unit_rate, conf_ecu_rate) * conf_query_weight)) / total_weight_cold\n ) * COALESCE(conf_chargeable_unit_rate, conf_ecu_rate)\n| STATS\n `Normalized cost` = SUM(blended) BY @timestamp, tier\n| WHERE `Normalized cost` > 0\n| RENAME tier AS `Data Tier`\n| SORT @timestamp ASC" - }, - "timeField": "@timestamp" - } - } - } - }, - "filters": [], - "needsRefresh": false, - "query": { - "esql": "FROM billing_realized_pool_lookup\n| LOOKUP JOIN cluster_capacity_utilization_lookup ON composite_key\n| LOOKUP JOIN cluster_deployment_contribution_lookup ON composite_key\n| LOOKUP JOIN cluster_tier_contribution_lookup ON composite_key\n| LOOKUP JOIN chargeback_conf_lookup ON @timestamp >= conf_start_date AND @timestamp <= conf_end_date\n| EVAL heap_util = COALESCE(heap_used_pct_p95, 100) / 100, disk_util = COALESCE(disk_used_pct_p95, 100) / 100, w_sum = conf_utilization_memory_weight + conf_utilization_storage_weight, capacity_utilization_score = CASE(w_sum > 0, (conf_utilization_memory_weight * heap_util + conf_utilization_storage_weight * disk_util) / w_sum, 1.0)\n| EVAL chargeable_pool = data_tier_capacity_ecu * COALESCE(capacity_utilization_score, 1.0), utilization_data_missing = heap_used_pct_p95 IS NULL AND disk_used_pct_p95 IS NULL\n| EVAL\n indexing = CASE (deployment_sum_indexing_time > 0, TO_DOUBLE(tier_sum_indexing_time) / deployment_sum_indexing_time * chargeable_pool) * COALESCE(conf_chargeable_unit_rate, conf_ecu_rate),\n querying = CASE (deployment_sum_query_time > 0, TO_DOUBLE(tier_sum_query_time) / deployment_sum_query_time * chargeable_pool) * COALESCE(conf_chargeable_unit_rate, conf_ecu_rate),\n data_set = CASE (deployment_sum_data_set_store_size > 0, TO_DOUBLE(tier_sum_data_set_store_size) / deployment_sum_data_set_store_size * chargeable_pool),\n store = CASE (deployment_sum_store_size > 0, TO_DOUBLE(tier_sum_store_size) / deployment_sum_store_size * chargeable_pool),\n storage = CASE (store == 0, data_set, store) * COALESCE(conf_chargeable_unit_rate, conf_ecu_rate),\n total_weight_hot = conf_storage_weight + conf_query_weight + conf_indexing_weight,\n total_weight_cold = conf_storage_weight + conf_query_weight,\n blended = CASE (\n tier == \"hot/content\",\n ((storage / COALESCE(conf_chargeable_unit_rate, conf_ecu_rate) * conf_storage_weight) + (querying / COALESCE(conf_chargeable_unit_rate, conf_ecu_rate) * conf_query_weight) + (indexing / COALESCE(conf_chargeable_unit_rate, conf_ecu_rate) * conf_indexing_weight)) / total_weight_hot,\n ((storage / COALESCE(conf_chargeable_unit_rate, conf_ecu_rate) * conf_storage_weight) + (querying / COALESCE(conf_chargeable_unit_rate, conf_ecu_rate) * conf_query_weight)) / total_weight_cold\n ) * COALESCE(conf_chargeable_unit_rate, conf_ecu_rate)\n| STATS\n `Normalized cost` = SUM(blended) BY @timestamp, tier\n| WHERE `Normalized cost` > 0\n| RENAME tier AS `Data Tier`\n| SORT @timestamp ASC" - }, - "visualization": { - "axisTitlesVisibilitySettings": { - "x": true, - "yLeft": true, - "yRight": true - }, - "fittingFunction": "Linear", - "gridlinesVisibilitySettings": { - "x": true, - "yLeft": true, - "yRight": true - }, - "labelsOrientation": { - "x": 0, - "yLeft": 0, - "yRight": 0 - }, - "layers": [ - { - "accessors": [ - "Normalized cost" - ], - "colorMapping": { - "assignments": [], - "colorMode": { - "type": "categorical" - }, - "paletteId": "eui_amsterdam_color_blind", - "specialAssignments": [ - { - "color": { - "type": "loop" - }, - "rules": [ - { - "type": "other" - } - ], - "touched": false - } - ] - }, - "layerId": "43f9cfe6-d828-4773-ab65-129f92cf7f4d", - "layerType": "data", - "seriesType": "bar_stacked", - "splitAccessor": "Data Tier", - "splitAccessors": [ - "Data Tier" - ], - "xAccessor": "@timestamp" - } - ], - "legend": { - "isVisible": false, - "position": "right", - "showSingleSeries": false - }, - "preferredSeriesType": "bar_stacked", - "tickLabelsVisibilitySettings": { - "x": true, - "yLeft": true, - "yRight": true - }, - "valueLabels": "hide" + "title": "Query Cost by Tier over Time", + "version": 2, + "visualizationType": "lnsXY" + }, + "title": "Query Cost by Tier over Time" + }, + "gridData": { + "h": 14, + "i": "fb59beb6-bf05-4ce0-90a3-d8b434d300aa", + "sectionId": "7f7cc9b5-13b3-4962-98f6-5b18a1d8ca80", + "w": 16, + "x": 16, + "y": 23 + }, + "panelIndex": "fb59beb6-bf05-4ce0-90a3-d8b434d300aa", + "type": "lens" + }, + { + "embeddableConfig": { + "attributes": { + "references": [], + "state": { + "datasourceStates": { + "textBased": { + "indexPatternRefs": [], + "layers": { + "91bcc41e-3a2c-434d-8df7-29c01340e712": { + "columns": [ + { + "columnId": "agg_querying", + "customLabel": true, + "fieldName": "agg_querying", + "inMetricDimension": true, + "label": "% Normalized cost", + "meta": { + "esType": "double", + "type": "number" + }, + "params": { + "format": { + "id": "number", + "params": { + "decimals": 2 + } } + } + }, + { + "columnId": "@timestamp", + "customLabel": true, + "fieldName": "@timestamp", + "label": "Date", + "meta": { + "esType": "date", + "type": "date" + } }, - "title": "Blended Cost by Tier over Time", - "version": 2, - "visualizationType": "lnsXY" + { + "columnId": "datastream", + "customLabel": true, + "fieldName": "datastream", + "label": "Data stream", + "meta": { + "esType": "keyword", + "type": "string" + } + } + ], + "index": "chargeback_integration", + "query": { + "esql": "FROM billing_realized_pool_lookup\n| LOOKUP JOIN cluster_capacity_utilization_lookup ON composite_key\n| LOOKUP JOIN cluster_deployment_contribution_lookup ON composite_key\n| LOOKUP JOIN cluster_datastream_contribution_lookup ON composite_key\n| LOOKUP JOIN chargeback_conf_lookup ON @timestamp >= conf_start_date AND @timestamp <= conf_end_date\n| EVAL heap_util = COALESCE(heap_used_pct_p95, 100) / 100, disk_util = COALESCE(disk_used_pct_p95, 100) / 100, w_sum = conf_utilization_memory_weight + conf_utilization_storage_weight, capacity_utilization_score = CASE(w_sum > 0, (conf_utilization_memory_weight * heap_util + conf_utilization_storage_weight * disk_util) / w_sum, 1.0)\n| EVAL chargeable_pool = data_tier_capacity_ecu * COALESCE(capacity_utilization_score, 1.0), utilization_data_missing = heap_used_pct_p95 IS NULL AND disk_used_pct_p95 IS NULL\n| EVAL \n querying = CASE (deployment_sum_query_time > 0, TO_DOUBLE(datastream_sum_query_time) / deployment_sum_query_time * chargeable_pool) * COALESCE(conf_chargeable_unit_rate, conf_ecu_rate)\n| STATS \n agg_querying = sum(querying)\n BY \n @timestamp, \n datastream\n| WHERE agg_querying > 0\n| SORT agg_querying DESC" + }, + "timeField": "@timestamp" + } + } + } + }, + "filters": [], + "needsRefresh": false, + "query": { + "esql": "FROM billing_realized_pool_lookup\n| LOOKUP JOIN cluster_capacity_utilization_lookup ON composite_key\n| LOOKUP JOIN cluster_deployment_contribution_lookup ON composite_key\n| LOOKUP JOIN cluster_datastream_contribution_lookup ON composite_key\n| LOOKUP JOIN chargeback_conf_lookup ON @timestamp >= conf_start_date AND @timestamp <= conf_end_date\n| EVAL heap_util = COALESCE(heap_used_pct_p95, 100) / 100, disk_util = COALESCE(disk_used_pct_p95, 100) / 100, w_sum = conf_utilization_memory_weight + conf_utilization_storage_weight, capacity_utilization_score = CASE(w_sum > 0, (conf_utilization_memory_weight * heap_util + conf_utilization_storage_weight * disk_util) / w_sum, 1.0)\n| EVAL chargeable_pool = data_tier_capacity_ecu * COALESCE(capacity_utilization_score, 1.0), utilization_data_missing = heap_used_pct_p95 IS NULL AND disk_used_pct_p95 IS NULL\n| EVAL \n querying = CASE (deployment_sum_query_time > 0, TO_DOUBLE(datastream_sum_query_time) / deployment_sum_query_time * chargeable_pool) * COALESCE(conf_chargeable_unit_rate, conf_ecu_rate)\n| STATS \n agg_querying = sum(querying)\n BY \n @timestamp, \n datastream\n| WHERE agg_querying > 0\n| SORT agg_querying DESC" + }, + "visualization": { + "axisTitlesVisibilitySettings": { + "x": true, + "yLeft": true, + "yRight": true + }, + "fittingFunction": "Linear", + "gridlinesVisibilitySettings": { + "x": true, + "yLeft": true, + "yRight": true + }, + "labelsOrientation": { + "x": 0, + "yLeft": 0, + "yRight": 0 + }, + "layers": [ + { + "accessors": [ + "agg_querying" + ], + "colorMapping": { + "assignments": [], + "colorMode": { + "type": "categorical" + }, + "paletteId": "eui_amsterdam_color_blind", + "specialAssignments": [ + { + "color": { + "type": "loop" + }, + "rules": [ + { + "type": "other" + } + ], + "touched": false + } + ] }, - "title": "Blended Cost by Tier over Time" - }, - "gridData": { - "h": 14, - "i": "b3e1e89a-a86d-40b3-97e2-ba89751330fa", - "sectionId": "7f7cc9b5-13b3-4962-98f6-5b18a1d8ca80", - "w": 16, - "x": 16, - "y": 58 - }, - "panelIndex": "b3e1e89a-a86d-40b3-97e2-ba89751330fa", - "type": "lens" + "layerId": "91bcc41e-3a2c-434d-8df7-29c01340e712", + "layerType": "data", + "seriesType": "bar_percentage_stacked", + "splitAccessor": "datastream", + "splitAccessors": [ + "datastream" + ], + "xAccessor": "@timestamp" + } + ], + "legend": { + "isVisible": false, + "position": "right", + "showSingleSeries": false + }, + "preferredSeriesType": "bar_stacked", + "tickLabelsVisibilitySettings": { + "x": true, + "yLeft": true, + "yRight": true + }, + "valueLabels": "hide" + } }, - { - "embeddableConfig": { - "attributes": { - "references": [], - "state": { - "datasourceStates": { - "textBased": { - "indexPatternRefs": [], - "layers": { - "d720a72f-fe61-4baf-bdee-8a50d4247670": { - "columns": [ - { - "columnId": "% Normalized cost", - "customLabel": false, - "fieldName": "% Normalized cost", - "inMetricDimension": true, - "label": "% Normalized cost", - "meta": { - "esType": "double", - "type": "number" - } - }, - { - "columnId": "@timestamp", - "customLabel": false, - "fieldName": "@timestamp", - "label": "@timestamp", - "meta": { - "esType": "date", - "type": "date" - } - }, - { - "columnId": "datastream", - "customLabel": false, - "fieldName": "datastream", - "label": "datastream", - "meta": { - "esType": "keyword", - "type": "string" - } - } - ], - "index": "chargeback_integration", - "query": { - "esql": "FROM billing_realized_pool_lookup\n| LOOKUP JOIN cluster_capacity_utilization_lookup ON composite_key\n| LOOKUP JOIN cluster_deployment_contribution_lookup ON composite_key\n| LOOKUP JOIN cluster_tier_and_datastream_contribution_lookup ON composite_key\n| LOOKUP JOIN chargeback_conf_lookup ON @timestamp >= conf_start_date AND @timestamp <= conf_end_date\n| EVAL heap_util = COALESCE(heap_used_pct_p95, 100) / 100, disk_util = COALESCE(disk_used_pct_p95, 100) / 100, w_sum = conf_utilization_memory_weight + conf_utilization_storage_weight, capacity_utilization_score = CASE(w_sum > 0, (conf_utilization_memory_weight * heap_util + conf_utilization_storage_weight * disk_util) / w_sum, 1.0)\n| EVAL chargeable_pool = data_tier_capacity_ecu * COALESCE(capacity_utilization_score, 1.0), utilization_data_missing = heap_used_pct_p95 IS NULL AND disk_used_pct_p95 IS NULL\n| EVAL\n indexing = CASE (deployment_sum_indexing_time > 0, TO_DOUBLE(tier_and_datastream_sum_indexing_time) / deployment_sum_indexing_time * chargeable_pool) * COALESCE(conf_chargeable_unit_rate, conf_ecu_rate),\n querying = CASE (deployment_sum_query_time > 0, TO_DOUBLE(tier_and_datastream_sum_query_time) / deployment_sum_query_time * chargeable_pool) * COALESCE(conf_chargeable_unit_rate, conf_ecu_rate),\n data_set = CASE (deployment_sum_data_set_store_size > 0, TO_DOUBLE(tier_and_datastream_sum_data_set_store_size) / deployment_sum_data_set_store_size * chargeable_pool),\n store = CASE (deployment_sum_store_size > 0, TO_DOUBLE(tier_and_datastream_sum_store_size) / deployment_sum_store_size * chargeable_pool),\n storage = CASE (store == 0, data_set, store) * COALESCE(conf_chargeable_unit_rate, conf_ecu_rate),\n total_weight_hot = conf_storage_weight + conf_query_weight + conf_indexing_weight,\n total_weight_cold = conf_storage_weight + conf_query_weight,\n blended = CASE (\n tier == \"hot/content\",\n ((storage / COALESCE(conf_chargeable_unit_rate, conf_ecu_rate) * conf_storage_weight) + (querying / COALESCE(conf_chargeable_unit_rate, conf_ecu_rate) * conf_query_weight) + (indexing / COALESCE(conf_chargeable_unit_rate, conf_ecu_rate) * conf_indexing_weight)) / total_weight_hot,\n ((storage / COALESCE(conf_chargeable_unit_rate, conf_ecu_rate) * conf_storage_weight) + (querying / COALESCE(conf_chargeable_unit_rate, conf_ecu_rate) * conf_query_weight)) / total_weight_cold\n ) * COALESCE(conf_chargeable_unit_rate, conf_ecu_rate)\n| STATS\n `% Normalized cost` = sum(blended)\n BY\n @timestamp,\n datastream,\n tier\n| WHERE `% Normalized cost` > 0\n| SORT `% Normalized cost` DESC" - }, - "timeField": "@timestamp" - } - } - } - }, - "filters": [], - "needsRefresh": false, - "query": { - "esql": "FROM billing_realized_pool_lookup\n| LOOKUP JOIN cluster_capacity_utilization_lookup ON composite_key\n| LOOKUP JOIN cluster_deployment_contribution_lookup ON composite_key\n| LOOKUP JOIN cluster_tier_and_datastream_contribution_lookup ON composite_key\n| LOOKUP JOIN chargeback_conf_lookup ON @timestamp >= conf_start_date AND @timestamp <= conf_end_date\n| EVAL heap_util = COALESCE(heap_used_pct_p95, 100) / 100, disk_util = COALESCE(disk_used_pct_p95, 100) / 100, w_sum = conf_utilization_memory_weight + conf_utilization_storage_weight, capacity_utilization_score = CASE(w_sum > 0, (conf_utilization_memory_weight * heap_util + conf_utilization_storage_weight * disk_util) / w_sum, 1.0)\n| EVAL chargeable_pool = data_tier_capacity_ecu * COALESCE(capacity_utilization_score, 1.0), utilization_data_missing = heap_used_pct_p95 IS NULL AND disk_used_pct_p95 IS NULL\n| EVAL\n indexing = CASE (deployment_sum_indexing_time > 0, TO_DOUBLE(tier_and_datastream_sum_indexing_time) / deployment_sum_indexing_time * chargeable_pool) * COALESCE(conf_chargeable_unit_rate, conf_ecu_rate),\n querying = CASE (deployment_sum_query_time > 0, TO_DOUBLE(tier_and_datastream_sum_query_time) / deployment_sum_query_time * chargeable_pool) * COALESCE(conf_chargeable_unit_rate, conf_ecu_rate),\n data_set = CASE (deployment_sum_data_set_store_size > 0, TO_DOUBLE(tier_and_datastream_sum_data_set_store_size) / deployment_sum_data_set_store_size * chargeable_pool),\n store = CASE (deployment_sum_store_size > 0, TO_DOUBLE(tier_and_datastream_sum_store_size) / deployment_sum_store_size * chargeable_pool),\n storage = CASE (store == 0, data_set, store) * COALESCE(conf_chargeable_unit_rate, conf_ecu_rate),\n total_weight_hot = conf_storage_weight + conf_query_weight + conf_indexing_weight,\n total_weight_cold = conf_storage_weight + conf_query_weight,\n blended = CASE (\n tier == \"hot/content\",\n ((storage / COALESCE(conf_chargeable_unit_rate, conf_ecu_rate) * conf_storage_weight) + (querying / COALESCE(conf_chargeable_unit_rate, conf_ecu_rate) * conf_query_weight) + (indexing / COALESCE(conf_chargeable_unit_rate, conf_ecu_rate) * conf_indexing_weight)) / total_weight_hot,\n ((storage / COALESCE(conf_chargeable_unit_rate, conf_ecu_rate) * conf_storage_weight) + (querying / COALESCE(conf_chargeable_unit_rate, conf_ecu_rate) * conf_query_weight)) / total_weight_cold\n ) * COALESCE(conf_chargeable_unit_rate, conf_ecu_rate)\n| STATS\n `% Normalized cost` = sum(blended)\n BY\n @timestamp,\n datastream,\n tier\n| WHERE `% Normalized cost` > 0\n| SORT `% Normalized cost` DESC" - }, - "visualization": { - "axisTitlesVisibilitySettings": { - "x": true, - "yLeft": true, - "yRight": true - }, - "fittingFunction": "Linear", - "gridlinesVisibilitySettings": { - "x": true, - "yLeft": true, - "yRight": true - }, - "labelsOrientation": { - "x": 0, - "yLeft": 0, - "yRight": 0 - }, - "layers": [ - { - "accessors": [ - "% Normalized cost" - ], - "colorMapping": { - "assignments": [], - "colorMode": { - "type": "categorical" - }, - "paletteId": "default", - "specialAssignments": [ - { - "color": { - "type": "loop" - }, - "rules": [ - { - "type": "other" - } - ], - "touched": false - } - ] - }, - "layerId": "d720a72f-fe61-4baf-bdee-8a50d4247670", - "layerType": "data", - "seriesType": "bar_percentage_stacked", - "splitAccessor": "datastream", - "splitAccessors": [ - "datastream" - ], - "xAccessor": "@timestamp" - } - ], - "legend": { - "isVisible": false, - "position": "right", - "showSingleSeries": false - }, - "preferredSeriesType": "bar", - "tickLabelsVisibilitySettings": { - "x": true, - "yLeft": true, - "yRight": true - }, - "valueLabels": "hide" + "title": "Query Cost Share by Data Stream over Time (%)", + "version": 2, + "visualizationType": "lnsXY" + }, + "title": "Query Cost Share by Data Stream over Time (%)" + }, + "gridData": { + "h": 14, + "i": "6fb65bcc-0568-4c8d-a833-c87cf8f8bccd", + "sectionId": "7f7cc9b5-13b3-4962-98f6-5b18a1d8ca80", + "w": 16, + "x": 32, + "y": 23 + }, + "panelIndex": "6fb65bcc-0568-4c8d-a833-c87cf8f8bccd", + "type": "lens" + }, + { + "embeddableConfig": { + "attributes": { + "references": [], + "state": { + "datasourceStates": { + "textBased": { + "indexPatternRefs": [], + "layers": { + "b72a0ab5-dbf5-4fd2-bdef-c855fe354bee": { + "columns": [ + { + "columnId": "agg_storage", + "customLabel": true, + "fieldName": "agg_storage", + "inMetricDimension": true, + "label": "Normalized cost", + "meta": { + "esType": "double", + "type": "number" + }, + "params": { + "format": { + "id": "number", + "params": { + "decimals": 2 + } } + } }, - "title": "Blended Cost Share by Tier and Data Stream (%)", - "version": 2, - "visualizationType": "lnsXY" - }, - "title": "Blended Cost Share by Tier and Data Stream (%)" - }, - "gridData": { - "h": 14, - "i": "cefe7754-4e91-4b85-8132-512a9ca31793", - "sectionId": "7f7cc9b5-13b3-4962-98f6-5b18a1d8ca80", - "w": 16, - "x": 32, - "y": 58 - }, - "panelIndex": "cefe7754-4e91-4b85-8132-512a9ca31793", - "type": "lens" - }, - { - "embeddableConfig": { - "enhancements": { - "dynamicActions": { - "events": [] + { + "columnId": "@timestamp", + "customLabel": true, + "fieldName": "@timestamp", + "label": "Date", + "meta": { + "esType": "date", + "type": "date" + } + }, + { + "columnId": "datastream", + "customLabel": true, + "fieldName": "datastream", + "label": "Data stream", + "meta": { + "esType": "keyword", + "type": "string" + } } + ], + "index": "chargeback_integration", + "query": { + "esql": "FROM billing_realized_pool_lookup\n| LOOKUP JOIN cluster_capacity_utilization_lookup ON composite_key\n| LOOKUP JOIN cluster_deployment_contribution_lookup ON composite_key\n| LOOKUP JOIN cluster_datastream_contribution_lookup ON composite_key\n| LOOKUP JOIN chargeback_conf_lookup ON @timestamp >= conf_start_date AND @timestamp <= conf_end_date\n| EVAL heap_util = COALESCE(heap_used_pct_p95, 100) / 100, disk_util = COALESCE(disk_used_pct_p95, 100) / 100, w_sum = conf_utilization_memory_weight + conf_utilization_storage_weight, capacity_utilization_score = CASE(w_sum > 0, (conf_utilization_memory_weight * heap_util + conf_utilization_storage_weight * disk_util) / w_sum, 1.0)\n| EVAL chargeable_pool = data_tier_capacity_ecu * COALESCE(capacity_utilization_score, 1.0), utilization_data_missing = heap_used_pct_p95 IS NULL AND disk_used_pct_p95 IS NULL\n| EVAL \n data_set = CASE (deployment_sum_data_set_store_size > 0, TO_DOUBLE(datastream_sum_data_set_store_size) / deployment_sum_data_set_store_size * chargeable_pool),\n store = CASE (deployment_sum_store_size > 0, TO_DOUBLE(datastream_sum_store_size) / deployment_sum_store_size * chargeable_pool),\n storage = CASE (store == 0, data_set, store) * COALESCE(conf_chargeable_unit_rate, conf_ecu_rate)\n| STATS \n agg_storage = sum(storage)\n BY \n @timestamp, \n datastream\n| WHERE agg_storage > 0\n| SORT agg_storage DESC" + }, + "timeField": "@timestamp" + } + } + } + }, + "filters": [], + "needsRefresh": false, + "query": { + "esql": "FROM billing_realized_pool_lookup\n| LOOKUP JOIN cluster_capacity_utilization_lookup ON composite_key\n| LOOKUP JOIN cluster_deployment_contribution_lookup ON composite_key\n| LOOKUP JOIN cluster_datastream_contribution_lookup ON composite_key\n| LOOKUP JOIN chargeback_conf_lookup ON @timestamp >= conf_start_date AND @timestamp <= conf_end_date\n| EVAL heap_util = COALESCE(heap_used_pct_p95, 100) / 100, disk_util = COALESCE(disk_used_pct_p95, 100) / 100, w_sum = conf_utilization_memory_weight + conf_utilization_storage_weight, capacity_utilization_score = CASE(w_sum > 0, (conf_utilization_memory_weight * heap_util + conf_utilization_storage_weight * disk_util) / w_sum, 1.0)\n| EVAL chargeable_pool = data_tier_capacity_ecu * COALESCE(capacity_utilization_score, 1.0), utilization_data_missing = heap_used_pct_p95 IS NULL AND disk_used_pct_p95 IS NULL\n| EVAL \n data_set = CASE (deployment_sum_data_set_store_size > 0, TO_DOUBLE(datastream_sum_data_set_store_size) / deployment_sum_data_set_store_size * chargeable_pool),\n store = CASE (deployment_sum_store_size > 0, TO_DOUBLE(datastream_sum_store_size) / deployment_sum_store_size * chargeable_pool),\n storage = CASE (store == 0, data_set, store) * COALESCE(conf_chargeable_unit_rate, conf_ecu_rate)\n| STATS \n agg_storage = sum(storage)\n BY \n @timestamp, \n datastream\n| WHERE agg_storage > 0\n| SORT agg_storage DESC" + }, + "visualization": { + "axisTitlesVisibilitySettings": { + "x": true, + "yLeft": true, + "yRight": true + }, + "fittingFunction": "Linear", + "gridlinesVisibilitySettings": { + "x": true, + "yLeft": true, + "yRight": true + }, + "labelsOrientation": { + "x": 0, + "yLeft": 0, + "yRight": 0 + }, + "layers": [ + { + "accessors": [ + "agg_storage" + ], + "colorMapping": { + "assignments": [], + "colorMode": { + "type": "categorical" + }, + "paletteId": "eui_amsterdam_color_blind", + "specialAssignments": [ + { + "color": { + "type": "loop" + }, + "rules": [ + { + "type": "other" + } + ], + "touched": false + } + ] }, - "savedVis": { - "data": { - "aggs": [], - "searchSource": { - "filter": [], - "query": { - "language": "kuery", - "query": "" - } + "layerId": "b72a0ab5-dbf5-4fd2-bdef-c855fe354bee", + "layerType": "data", + "seriesType": "bar_stacked", + "splitAccessor": "datastream", + "splitAccessors": [ + "datastream" + ], + "xAccessor": "@timestamp" + } + ], + "legend": { + "isVisible": false, + "position": "right", + "showSingleSeries": false + }, + "preferredSeriesType": "bar_stacked", + "tickLabelsVisibilitySettings": { + "x": true, + "yLeft": true, + "yRight": true + }, + "valueLabels": "hide" + } + }, + "title": "Storage Cost by Data Stream over Time", + "version": 2, + "visualizationType": "lnsXY" + }, + "title": "Storage Cost by Data Stream over Time" + }, + "gridData": { + "h": 15, + "i": "5bdd9079-cb0e-453e-835b-afcb5d72f06b", + "sectionId": "7f7cc9b5-13b3-4962-98f6-5b18a1d8ca80", + "w": 16, + "x": 0, + "y": 40 + }, + "panelIndex": "5bdd9079-cb0e-453e-835b-afcb5d72f06b", + "type": "lens" + }, + { + "embeddableConfig": { + "attributes": { + "references": [], + "state": { + "datasourceStates": { + "textBased": { + "indexPatternRefs": [], + "layers": { + "93fd5f12-463f-4e8e-a9b8-0ac77d0522a9": { + "columns": [ + { + "columnId": "agg_storage", + "customLabel": true, + "fieldName": "agg_storage", + "inMetricDimension": true, + "label": "Normalized cost", + "meta": { + "esType": "double", + "type": "number" + }, + "params": { + "format": { + "id": "number", + "params": { + "decimals": 2 + } } + } }, - "description": "", - "id": "", - "params": { - "fontSize": 12, - "markdown": "# Data tier and data stream per day", - "openLinksInNewTab": false + { + "columnId": "@timestamp", + "customLabel": true, + "fieldName": "@timestamp", + "label": "Date", + "meta": { + "esType": "date", + "type": "date" + } }, - "title": "", - "type": "markdown", - "uiState": {} + { + "columnId": "tier", + "customLabel": true, + "fieldName": "tier", + "label": "Data Tier", + "meta": { + "esType": "keyword", + "type": "string" + } + } + ], + "index": "chargeback_integration", + "query": { + "esql": "FROM billing_realized_pool_lookup\n| LOOKUP JOIN cluster_capacity_utilization_lookup ON composite_key\n| LOOKUP JOIN cluster_deployment_contribution_lookup ON composite_key\n| LOOKUP JOIN cluster_tier_contribution_lookup ON composite_key\n| LOOKUP JOIN chargeback_conf_lookup ON @timestamp >= conf_start_date AND @timestamp <= conf_end_date\n| EVAL heap_util = COALESCE(heap_used_pct_p95, 100) / 100, disk_util = COALESCE(disk_used_pct_p95, 100) / 100, w_sum = conf_utilization_memory_weight + conf_utilization_storage_weight, capacity_utilization_score = CASE(w_sum > 0, (conf_utilization_memory_weight * heap_util + conf_utilization_storage_weight * disk_util) / w_sum, 1.0)\n| EVAL chargeable_pool = data_tier_capacity_ecu * COALESCE(capacity_utilization_score, 1.0), utilization_data_missing = heap_used_pct_p95 IS NULL AND disk_used_pct_p95 IS NULL\n| EVAL \n data_set = CASE (deployment_sum_data_set_store_size > 0, TO_DOUBLE(tier_sum_data_set_store_size) / deployment_sum_data_set_store_size * chargeable_pool),\n store = CASE (deployment_sum_store_size > 0, TO_DOUBLE(tier_sum_store_size) / deployment_sum_store_size * chargeable_pool),\n storage = CASE (store == 0, data_set, store) * COALESCE(conf_chargeable_unit_rate, conf_ecu_rate)\n| STATS \n agg_storage = sum(storage)\n BY \n @timestamp,\n tier\n| WHERE agg_storage > 0" + }, + "timeField": "@timestamp" } - }, - "gridData": { - "h": 3, - "i": "5994f908-7b57-4bd9-8e58-0b275c4c7f8e", - "sectionId": "7f7cc9b5-13b3-4962-98f6-5b18a1d8ca80", - "w": 48, - "x": 0, - "y": 0 - }, - "panelIndex": "5994f908-7b57-4bd9-8e58-0b275c4c7f8e", - "type": "visualization" - }, - { - "embeddableConfig": { - "enhancements": { - "dynamicActions": { - "events": [] + } + } + }, + "filters": [], + "needsRefresh": false, + "query": { + "esql": "FROM billing_realized_pool_lookup\n| LOOKUP JOIN cluster_capacity_utilization_lookup ON composite_key\n| LOOKUP JOIN cluster_deployment_contribution_lookup ON composite_key\n| LOOKUP JOIN cluster_tier_contribution_lookup ON composite_key\n| LOOKUP JOIN chargeback_conf_lookup ON @timestamp >= conf_start_date AND @timestamp <= conf_end_date\n| EVAL heap_util = COALESCE(heap_used_pct_p95, 100) / 100, disk_util = COALESCE(disk_used_pct_p95, 100) / 100, w_sum = conf_utilization_memory_weight + conf_utilization_storage_weight, capacity_utilization_score = CASE(w_sum > 0, (conf_utilization_memory_weight * heap_util + conf_utilization_storage_weight * disk_util) / w_sum, 1.0)\n| EVAL chargeable_pool = data_tier_capacity_ecu * COALESCE(capacity_utilization_score, 1.0), utilization_data_missing = heap_used_pct_p95 IS NULL AND disk_used_pct_p95 IS NULL\n| EVAL \n data_set = CASE (deployment_sum_data_set_store_size > 0, TO_DOUBLE(tier_sum_data_set_store_size) / deployment_sum_data_set_store_size * chargeable_pool),\n store = CASE (deployment_sum_store_size > 0, TO_DOUBLE(tier_sum_store_size) / deployment_sum_store_size * chargeable_pool),\n storage = CASE (store == 0, data_set, store) * COALESCE(conf_chargeable_unit_rate, conf_ecu_rate)\n| STATS \n agg_storage = sum(storage)\n BY \n @timestamp,\n tier\n| WHERE agg_storage > 0" + }, + "visualization": { + "axisTitlesVisibilitySettings": { + "x": true, + "yLeft": true, + "yRight": true + }, + "fittingFunction": "Linear", + "gridlinesVisibilitySettings": { + "x": true, + "yLeft": true, + "yRight": true + }, + "labelsOrientation": { + "x": 0, + "yLeft": 0, + "yRight": 0 + }, + "layers": [ + { + "accessors": [ + "agg_storage" + ], + "colorMapping": { + "assignments": [], + "colorMode": { + "type": "categorical" + }, + "paletteId": "eui_amsterdam_color_blind", + "specialAssignments": [ + { + "color": { + "type": "loop" + }, + "rules": [ + { + "type": "other" + } + ], + "touched": false } + ] }, - "savedVis": { - "data": { - "aggs": [], - "searchSource": { - "filter": [], - "query": { - "language": "kuery", - "query": "" - } + "layerId": "93fd5f12-463f-4e8e-a9b8-0ac77d0522a9", + "layerType": "data", + "seriesType": "bar_stacked", + "splitAccessor": "tier", + "splitAccessors": [ + "tier" + ], + "xAccessor": "@timestamp" + } + ], + "legend": { + "isVisible": false, + "position": "right", + "showSingleSeries": false + }, + "preferredSeriesType": "bar_stacked", + "tickLabelsVisibilitySettings": { + "x": true, + "yLeft": true, + "yRight": true + }, + "valueLabels": "hide" + } + }, + "title": "Storage Cost by Tier over Time", + "version": 2, + "visualizationType": "lnsXY" + }, + "title": "Storage Cost by Tier over Time" + }, + "gridData": { + "h": 15, + "i": "5ac2dcd4-8775-4c15-8e79-346e0a62dd4d", + "sectionId": "7f7cc9b5-13b3-4962-98f6-5b18a1d8ca80", + "w": 16, + "x": 16, + "y": 40 + }, + "panelIndex": "5ac2dcd4-8775-4c15-8e79-346e0a62dd4d", + "type": "lens" + }, + { + "embeddableConfig": { + "attributes": { + "references": [], + "state": { + "datasourceStates": { + "textBased": { + "indexPatternRefs": [], + "layers": { + "b72a0ab5-dbf5-4fd2-bdef-c855fe354bee": { + "columns": [ + { + "columnId": "agg_storage", + "customLabel": true, + "fieldName": "agg_storage", + "inMetricDimension": true, + "label": "% Normalized cost", + "meta": { + "esType": "double", + "type": "number" + }, + "params": { + "format": { + "id": "number", + "params": { + "decimals": 2 + } } + } }, - "description": "", - "id": "", - "params": { - "fontSize": 12, - "markdown": "## Indexing details", - "openLinksInNewTab": false + { + "columnId": "@timestamp", + "customLabel": true, + "fieldName": "@timestamp", + "label": "Date", + "meta": { + "esType": "date", + "type": "date" + } }, - "title": "", - "type": "markdown", - "uiState": {} + { + "columnId": "datastream", + "customLabel": true, + "fieldName": "datastream", + "label": "Data stream", + "meta": { + "esType": "keyword", + "type": "string" + } + } + ], + "index": "chargeback_integration", + "query": { + "esql": "FROM billing_realized_pool_lookup\n| LOOKUP JOIN cluster_capacity_utilization_lookup ON composite_key\n| LOOKUP JOIN cluster_deployment_contribution_lookup ON composite_key\n| LOOKUP JOIN cluster_datastream_contribution_lookup ON composite_key\n| LOOKUP JOIN chargeback_conf_lookup ON @timestamp >= conf_start_date AND @timestamp <= conf_end_date\n| EVAL heap_util = COALESCE(heap_used_pct_p95, 100) / 100, disk_util = COALESCE(disk_used_pct_p95, 100) / 100, w_sum = conf_utilization_memory_weight + conf_utilization_storage_weight, capacity_utilization_score = CASE(w_sum > 0, (conf_utilization_memory_weight * heap_util + conf_utilization_storage_weight * disk_util) / w_sum, 1.0)\n| EVAL chargeable_pool = data_tier_capacity_ecu * COALESCE(capacity_utilization_score, 1.0), utilization_data_missing = heap_used_pct_p95 IS NULL AND disk_used_pct_p95 IS NULL\n| EVAL \n data_set = CASE (deployment_sum_data_set_store_size > 0, TO_DOUBLE(datastream_sum_data_set_store_size) / deployment_sum_data_set_store_size * chargeable_pool),\n store = CASE (deployment_sum_store_size > 0, TO_DOUBLE(datastream_sum_store_size) / deployment_sum_store_size * chargeable_pool),\n storage = CASE (store == 0, data_set, store) * COALESCE(conf_chargeable_unit_rate, conf_ecu_rate)\n| STATS \n agg_storage = sum(storage)\n BY \n @timestamp, \n datastream\n| WHERE agg_storage > 0\n| SORT agg_storage DESC" + }, + "timeField": "@timestamp" } - }, - "gridData": { - "h": 3, - "i": "5011c332-ce8a-4a78-a08a-c0527c374fcb", - "sectionId": "7f7cc9b5-13b3-4962-98f6-5b18a1d8ca80", - "w": 48, - "x": 0, - "y": 3 - }, - "panelIndex": "5011c332-ce8a-4a78-a08a-c0527c374fcb", - "type": "visualization" - }, - { - "embeddableConfig": { - "enhancements": { - "dynamicActions": { - "events": [] + } + } + }, + "filters": [], + "needsRefresh": false, + "query": { + "esql": "FROM billing_realized_pool_lookup\n| LOOKUP JOIN cluster_capacity_utilization_lookup ON composite_key\n| LOOKUP JOIN cluster_deployment_contribution_lookup ON composite_key\n| LOOKUP JOIN cluster_datastream_contribution_lookup ON composite_key\n| LOOKUP JOIN chargeback_conf_lookup ON @timestamp >= conf_start_date AND @timestamp <= conf_end_date\n| EVAL heap_util = COALESCE(heap_used_pct_p95, 100) / 100, disk_util = COALESCE(disk_used_pct_p95, 100) / 100, w_sum = conf_utilization_memory_weight + conf_utilization_storage_weight, capacity_utilization_score = CASE(w_sum > 0, (conf_utilization_memory_weight * heap_util + conf_utilization_storage_weight * disk_util) / w_sum, 1.0)\n| EVAL chargeable_pool = data_tier_capacity_ecu * COALESCE(capacity_utilization_score, 1.0), utilization_data_missing = heap_used_pct_p95 IS NULL AND disk_used_pct_p95 IS NULL\n| EVAL \n data_set = CASE (deployment_sum_data_set_store_size > 0, TO_DOUBLE(datastream_sum_data_set_store_size) / deployment_sum_data_set_store_size * chargeable_pool),\n store = CASE (deployment_sum_store_size > 0, TO_DOUBLE(datastream_sum_store_size) / deployment_sum_store_size * chargeable_pool),\n storage = CASE (store == 0, data_set, store) * COALESCE(conf_chargeable_unit_rate, conf_ecu_rate)\n| STATS \n agg_storage = sum(storage)\n BY \n @timestamp, \n datastream\n| WHERE agg_storage > 0\n| SORT agg_storage DESC" + }, + "visualization": { + "axisTitlesVisibilitySettings": { + "x": true, + "yLeft": true, + "yRight": true + }, + "fittingFunction": "Linear", + "gridlinesVisibilitySettings": { + "x": true, + "yLeft": true, + "yRight": true + }, + "labelsOrientation": { + "x": 0, + "yLeft": 0, + "yRight": 0 + }, + "layers": [ + { + "accessors": [ + "agg_storage" + ], + "colorMapping": { + "assignments": [], + "colorMode": { + "type": "categorical" + }, + "paletteId": "eui_amsterdam_color_blind", + "specialAssignments": [ + { + "color": { + "type": "loop" + }, + "rules": [ + { + "type": "other" + } + ], + "touched": false } + ] }, - "savedVis": { - "data": { - "aggs": [], - "searchSource": { - "filter": [], - "query": { - "language": "kuery", - "query": "" - } + "layerId": "b72a0ab5-dbf5-4fd2-bdef-c855fe354bee", + "layerType": "data", + "seriesType": "bar_percentage_stacked", + "splitAccessor": "datastream", + "splitAccessors": [ + "datastream" + ], + "xAccessor": "@timestamp" + } + ], + "legend": { + "isVisible": false, + "position": "right", + "showSingleSeries": false + }, + "preferredSeriesType": "bar_stacked", + "tickLabelsVisibilitySettings": { + "x": true, + "yLeft": true, + "yRight": true + }, + "valueLabels": "hide" + } + }, + "title": "Storage Cost Share by Data Stream over Time (%)", + "version": 2, + "visualizationType": "lnsXY" + }, + "title": "Storage Cost Share by Data Stream over Time (%)" + }, + "gridData": { + "h": 15, + "i": "5d96941f-c4e3-4833-b716-192d75d6ae0d", + "sectionId": "7f7cc9b5-13b3-4962-98f6-5b18a1d8ca80", + "w": 16, + "x": 32, + "y": 40 + }, + "panelIndex": "5d96941f-c4e3-4833-b716-192d75d6ae0d", + "type": "lens" + }, + { + "embeddableConfig": { + "attributes": { + "references": [], + "state": { + "datasourceStates": { + "textBased": { + "indexPatternRefs": [], + "layers": { + "52906f25-a114-4d45-b425-f0ff154f4ac1": { + "columns": [ + { + "columnId": "agg_blended", + "customLabel": true, + "fieldName": "agg_blended", + "inMetricDimension": true, + "label": "Normalized cost", + "meta": { + "esType": "double", + "type": "number" + }, + "params": { + "format": { + "id": "number", + "params": { + "decimals": 2 + } } + } }, - "description": "", - "id": "", - "params": { - "fontSize": 12, - "markdown": "## Querying details", - "openLinksInNewTab": false + { + "columnId": "@timestamp", + "customLabel": true, + "fieldName": "@timestamp", + "label": "Date", + "meta": { + "esType": "date", + "type": "date" + } }, - "title": "", - "type": "markdown", - "uiState": {} + { + "columnId": "datastream", + "customLabel": false, + "fieldName": "datastream", + "label": "datastream", + "meta": { + "esType": "keyword", + "type": "string" + } + } + ], + "index": "chargeback_integration", + "query": { + "esql": "FROM billing_realized_pool_lookup\n| LOOKUP JOIN cluster_capacity_utilization_lookup ON composite_key\n| LOOKUP JOIN cluster_deployment_contribution_lookup ON composite_key\n| LOOKUP JOIN cluster_datastream_contribution_lookup ON composite_key\n| LOOKUP JOIN chargeback_conf_lookup ON @timestamp >= conf_start_date AND @timestamp <= conf_end_date\n| EVAL heap_util = COALESCE(heap_used_pct_p95, 100) / 100, disk_util = COALESCE(disk_used_pct_p95, 100) / 100, w_sum = conf_utilization_memory_weight + conf_utilization_storage_weight, capacity_utilization_score = CASE(w_sum > 0, (conf_utilization_memory_weight * heap_util + conf_utilization_storage_weight * disk_util) / w_sum, 1.0)\n| EVAL chargeable_pool = data_tier_capacity_ecu * COALESCE(capacity_utilization_score, 1.0), utilization_data_missing = heap_used_pct_p95 IS NULL AND disk_used_pct_p95 IS NULL\n| EVAL\n indexing = CASE (deployment_sum_indexing_time > 0, TO_DOUBLE(datastream_sum_indexing_time) / deployment_sum_indexing_time * chargeable_pool),\n querying = CASE (deployment_sum_query_time > 0, TO_DOUBLE(datastream_sum_query_time) / deployment_sum_query_time * chargeable_pool),\n data_set = CASE (deployment_sum_data_set_store_size > 0, TO_DOUBLE(datastream_sum_data_set_store_size) / deployment_sum_data_set_store_size * chargeable_pool),\n store = CASE (deployment_sum_store_size > 0, TO_DOUBLE(datastream_sum_store_size) / deployment_sum_store_size * chargeable_pool),\n storage = CASE (store == 0, data_set, store),\n total_weight_hot = conf_storage_weight + conf_query_weight + conf_indexing_weight,\n blended = TO_DOUBLE(((storage * conf_storage_weight) + (querying * conf_query_weight) + (indexing * conf_indexing_weight))) / total_weight_hot * COALESCE(conf_chargeable_unit_rate, conf_ecu_rate)\n| STATS\n agg_blended = sum(blended)\n BY\n @timestamp,\n datastream\n| WHERE agg_blended > 0\n| SORT agg_blended DESC" + }, + "timeField": "@timestamp" } - }, - "gridData": { - "h": 3, - "i": "e458e034-197a-4ec6-95a8-2938181a4883", - "sectionId": "7f7cc9b5-13b3-4962-98f6-5b18a1d8ca80", - "w": 48, - "x": 0, - "y": 20 - }, - "panelIndex": "e458e034-197a-4ec6-95a8-2938181a4883", - "type": "visualization" - }, - { - "embeddableConfig": { - "enhancements": { - "dynamicActions": { - "events": [] + } + } + }, + "filters": [], + "needsRefresh": false, + "query": { + "esql": "FROM billing_realized_pool_lookup\n| LOOKUP JOIN cluster_capacity_utilization_lookup ON composite_key\n| LOOKUP JOIN cluster_deployment_contribution_lookup ON composite_key\n| LOOKUP JOIN cluster_datastream_contribution_lookup ON composite_key\n| LOOKUP JOIN chargeback_conf_lookup ON @timestamp >= conf_start_date AND @timestamp <= conf_end_date\n| EVAL heap_util = COALESCE(heap_used_pct_p95, 100) / 100, disk_util = COALESCE(disk_used_pct_p95, 100) / 100, w_sum = conf_utilization_memory_weight + conf_utilization_storage_weight, capacity_utilization_score = CASE(w_sum > 0, (conf_utilization_memory_weight * heap_util + conf_utilization_storage_weight * disk_util) / w_sum, 1.0)\n| EVAL chargeable_pool = data_tier_capacity_ecu * COALESCE(capacity_utilization_score, 1.0), utilization_data_missing = heap_used_pct_p95 IS NULL AND disk_used_pct_p95 IS NULL\n| EVAL\n indexing = CASE (deployment_sum_indexing_time > 0, TO_DOUBLE(datastream_sum_indexing_time) / deployment_sum_indexing_time * chargeable_pool),\n querying = CASE (deployment_sum_query_time > 0, TO_DOUBLE(datastream_sum_query_time) / deployment_sum_query_time * chargeable_pool),\n data_set = CASE (deployment_sum_data_set_store_size > 0, TO_DOUBLE(datastream_sum_data_set_store_size) / deployment_sum_data_set_store_size * chargeable_pool),\n store = CASE (deployment_sum_store_size > 0, TO_DOUBLE(datastream_sum_store_size) / deployment_sum_store_size * chargeable_pool),\n storage = CASE (store == 0, data_set, store),\n total_weight_hot = conf_storage_weight + conf_query_weight + conf_indexing_weight,\n blended = TO_DOUBLE(((storage * conf_storage_weight) + (querying * conf_query_weight) + (indexing * conf_indexing_weight))) / total_weight_hot * COALESCE(conf_chargeable_unit_rate, conf_ecu_rate)\n| STATS\n agg_blended = sum(blended)\n BY\n @timestamp,\n datastream\n| WHERE agg_blended > 0\n| SORT agg_blended DESC" + }, + "visualization": { + "axisTitlesVisibilitySettings": { + "x": true, + "yLeft": true, + "yRight": true + }, + "fittingFunction": "Linear", + "gridlinesVisibilitySettings": { + "x": true, + "yLeft": true, + "yRight": true + }, + "labelsOrientation": { + "x": 0, + "yLeft": 0, + "yRight": 0 + }, + "layers": [ + { + "accessors": [ + "agg_blended" + ], + "colorMapping": { + "assignments": [], + "colorMode": { + "type": "categorical" + }, + "paletteId": "eui_amsterdam_color_blind", + "specialAssignments": [ + { + "color": { + "type": "loop" + }, + "rules": [ + { + "type": "other" + } + ], + "touched": false } + ] }, - "savedVis": { - "data": { - "aggs": [], - "searchSource": { - "filter": [], - "query": { - "language": "kuery", - "query": "" - } - } + "layerId": "52906f25-a114-4d45-b425-f0ff154f4ac1", + "layerType": "data", + "seriesType": "bar_stacked", + "splitAccessor": "datastream", + "splitAccessors": [ + "datastream" + ], + "xAccessor": "@timestamp" + } + ], + "legend": { + "isVisible": false, + "position": "right", + "showSingleSeries": false + }, + "preferredSeriesType": "bar_stacked", + "tickLabelsVisibilitySettings": { + "x": true, + "yLeft": true, + "yRight": true + }, + "valueLabels": "hide" + } + }, + "title": "Blended Cost by Data Stream over Time", + "version": 2, + "visualizationType": "lnsXY" + }, + "title": "Blended Cost by Data Stream over Time" + }, + "gridData": { + "h": 14, + "i": "0f25691b-8bc9-4dd2-94be-893a6e64a195", + "sectionId": "7f7cc9b5-13b3-4962-98f6-5b18a1d8ca80", + "w": 16, + "x": 0, + "y": 58 + }, + "panelIndex": "0f25691b-8bc9-4dd2-94be-893a6e64a195", + "type": "lens" + }, + { + "embeddableConfig": { + "attributes": { + "references": [], + "state": { + "datasourceStates": { + "textBased": { + "indexPatternRefs": [], + "layers": { + "43f9cfe6-d828-4773-ab65-129f92cf7f4d": { + "columns": [ + { + "columnId": "@timestamp", + "customLabel": false, + "fieldName": "@timestamp", + "label": "@timestamp", + "meta": { + "esType": "date", + "type": "date" + } }, - "description": "", - "id": "", - "params": { - "fontSize": 12, - "markdown": "## Storage details", - "openLinksInNewTab": false + { + "columnId": "Data Tier", + "customLabel": false, + "fieldName": "Data Tier", + "label": "Data Tier", + "meta": { + "esType": "keyword", + "type": "string" + } }, - "title": "", - "type": "markdown", - "uiState": {} + { + "columnId": "Normalized cost", + "customLabel": false, + "fieldName": "Normalized cost", + "inMetricDimension": true, + "label": "Normalized cost", + "meta": { + "esType": "double", + "type": "number" + } + } + ], + "index": "chargeback_integration", + "query": { + "esql": "FROM billing_realized_pool_lookup\n| LOOKUP JOIN cluster_capacity_utilization_lookup ON composite_key\n| LOOKUP JOIN cluster_deployment_contribution_lookup ON composite_key\n| LOOKUP JOIN cluster_tier_contribution_lookup ON composite_key\n| LOOKUP JOIN chargeback_conf_lookup ON @timestamp >= conf_start_date AND @timestamp <= conf_end_date\n| EVAL heap_util = COALESCE(heap_used_pct_p95, 100) / 100, disk_util = COALESCE(disk_used_pct_p95, 100) / 100, w_sum = conf_utilization_memory_weight + conf_utilization_storage_weight, capacity_utilization_score = CASE(w_sum > 0, (conf_utilization_memory_weight * heap_util + conf_utilization_storage_weight * disk_util) / w_sum, 1.0)\n| EVAL chargeable_pool = data_tier_capacity_ecu * COALESCE(capacity_utilization_score, 1.0), utilization_data_missing = heap_used_pct_p95 IS NULL AND disk_used_pct_p95 IS NULL\n| EVAL\n indexing = CASE (deployment_sum_indexing_time > 0, TO_DOUBLE(tier_sum_indexing_time) / deployment_sum_indexing_time * chargeable_pool) * COALESCE(conf_chargeable_unit_rate, conf_ecu_rate),\n querying = CASE (deployment_sum_query_time > 0, TO_DOUBLE(tier_sum_query_time) / deployment_sum_query_time * chargeable_pool) * COALESCE(conf_chargeable_unit_rate, conf_ecu_rate),\n data_set = CASE (deployment_sum_data_set_store_size > 0, TO_DOUBLE(tier_sum_data_set_store_size) / deployment_sum_data_set_store_size * chargeable_pool),\n store = CASE (deployment_sum_store_size > 0, TO_DOUBLE(tier_sum_store_size) / deployment_sum_store_size * chargeable_pool),\n storage = CASE (store == 0, data_set, store) * COALESCE(conf_chargeable_unit_rate, conf_ecu_rate),\n total_weight_hot = conf_storage_weight + conf_query_weight + conf_indexing_weight,\n total_weight_cold = conf_storage_weight + conf_query_weight,\n blended = CASE (\n tier == \"hot/content\",\n ((storage / COALESCE(conf_chargeable_unit_rate, conf_ecu_rate) * conf_storage_weight) + (querying / COALESCE(conf_chargeable_unit_rate, conf_ecu_rate) * conf_query_weight) + (indexing / COALESCE(conf_chargeable_unit_rate, conf_ecu_rate) * conf_indexing_weight)) / total_weight_hot,\n ((storage / COALESCE(conf_chargeable_unit_rate, conf_ecu_rate) * conf_storage_weight) + (querying / COALESCE(conf_chargeable_unit_rate, conf_ecu_rate) * conf_query_weight)) / total_weight_cold\n ) * COALESCE(conf_chargeable_unit_rate, conf_ecu_rate)\n| STATS\n `Normalized cost` = SUM(blended) BY @timestamp, tier\n| WHERE `Normalized cost` > 0\n| RENAME tier AS `Data Tier`\n| SORT @timestamp ASC" + }, + "timeField": "@timestamp" } - }, - "gridData": { - "h": 3, - "i": "692b3cdd-b37f-4da9-80a6-aff7b799ecc1", - "sectionId": "7f7cc9b5-13b3-4962-98f6-5b18a1d8ca80", - "w": 48, - "x": 0, - "y": 37 - }, - "panelIndex": "692b3cdd-b37f-4da9-80a6-aff7b799ecc1", - "type": "visualization" - }, - { - "embeddableConfig": { - "enhancements": { - "dynamicActions": { - "events": [] + } + } + }, + "filters": [], + "needsRefresh": false, + "query": { + "esql": "FROM billing_realized_pool_lookup\n| LOOKUP JOIN cluster_capacity_utilization_lookup ON composite_key\n| LOOKUP JOIN cluster_deployment_contribution_lookup ON composite_key\n| LOOKUP JOIN cluster_tier_contribution_lookup ON composite_key\n| LOOKUP JOIN chargeback_conf_lookup ON @timestamp >= conf_start_date AND @timestamp <= conf_end_date\n| EVAL heap_util = COALESCE(heap_used_pct_p95, 100) / 100, disk_util = COALESCE(disk_used_pct_p95, 100) / 100, w_sum = conf_utilization_memory_weight + conf_utilization_storage_weight, capacity_utilization_score = CASE(w_sum > 0, (conf_utilization_memory_weight * heap_util + conf_utilization_storage_weight * disk_util) / w_sum, 1.0)\n| EVAL chargeable_pool = data_tier_capacity_ecu * COALESCE(capacity_utilization_score, 1.0), utilization_data_missing = heap_used_pct_p95 IS NULL AND disk_used_pct_p95 IS NULL\n| EVAL\n indexing = CASE (deployment_sum_indexing_time > 0, TO_DOUBLE(tier_sum_indexing_time) / deployment_sum_indexing_time * chargeable_pool) * COALESCE(conf_chargeable_unit_rate, conf_ecu_rate),\n querying = CASE (deployment_sum_query_time > 0, TO_DOUBLE(tier_sum_query_time) / deployment_sum_query_time * chargeable_pool) * COALESCE(conf_chargeable_unit_rate, conf_ecu_rate),\n data_set = CASE (deployment_sum_data_set_store_size > 0, TO_DOUBLE(tier_sum_data_set_store_size) / deployment_sum_data_set_store_size * chargeable_pool),\n store = CASE (deployment_sum_store_size > 0, TO_DOUBLE(tier_sum_store_size) / deployment_sum_store_size * chargeable_pool),\n storage = CASE (store == 0, data_set, store) * COALESCE(conf_chargeable_unit_rate, conf_ecu_rate),\n total_weight_hot = conf_storage_weight + conf_query_weight + conf_indexing_weight,\n total_weight_cold = conf_storage_weight + conf_query_weight,\n blended = CASE (\n tier == \"hot/content\",\n ((storage / COALESCE(conf_chargeable_unit_rate, conf_ecu_rate) * conf_storage_weight) + (querying / COALESCE(conf_chargeable_unit_rate, conf_ecu_rate) * conf_query_weight) + (indexing / COALESCE(conf_chargeable_unit_rate, conf_ecu_rate) * conf_indexing_weight)) / total_weight_hot,\n ((storage / COALESCE(conf_chargeable_unit_rate, conf_ecu_rate) * conf_storage_weight) + (querying / COALESCE(conf_chargeable_unit_rate, conf_ecu_rate) * conf_query_weight)) / total_weight_cold\n ) * COALESCE(conf_chargeable_unit_rate, conf_ecu_rate)\n| STATS\n `Normalized cost` = SUM(blended) BY @timestamp, tier\n| WHERE `Normalized cost` > 0\n| RENAME tier AS `Data Tier`\n| SORT @timestamp ASC" + }, + "visualization": { + "axisTitlesVisibilitySettings": { + "x": true, + "yLeft": true, + "yRight": true + }, + "fittingFunction": "Linear", + "gridlinesVisibilitySettings": { + "x": true, + "yLeft": true, + "yRight": true + }, + "labelsOrientation": { + "x": 0, + "yLeft": 0, + "yRight": 0 + }, + "layers": [ + { + "accessors": [ + "Normalized cost" + ], + "colorMapping": { + "assignments": [], + "colorMode": { + "type": "categorical" + }, + "paletteId": "eui_amsterdam_color_blind", + "specialAssignments": [ + { + "color": { + "type": "loop" + }, + "rules": [ + { + "type": "other" + } + ], + "touched": false } + ] }, - "savedVis": { - "data": { - "aggs": [], - "searchSource": { - "filter": [], - "query": { - "language": "kuery", - "query": "" - } - } + "layerId": "43f9cfe6-d828-4773-ab65-129f92cf7f4d", + "layerType": "data", + "seriesType": "bar_stacked", + "splitAccessor": "Data Tier", + "splitAccessors": [ + "Data Tier" + ], + "xAccessor": "@timestamp" + } + ], + "legend": { + "isVisible": false, + "position": "right", + "showSingleSeries": false + }, + "preferredSeriesType": "bar_stacked", + "tickLabelsVisibilitySettings": { + "x": true, + "yLeft": true, + "yRight": true + }, + "valueLabels": "hide" + } + }, + "title": "Blended Cost by Tier over Time", + "version": 2, + "visualizationType": "lnsXY" + }, + "title": "Blended Cost by Tier over Time" + }, + "gridData": { + "h": 14, + "i": "b3e1e89a-a86d-40b3-97e2-ba89751330fa", + "sectionId": "7f7cc9b5-13b3-4962-98f6-5b18a1d8ca80", + "w": 16, + "x": 16, + "y": 58 + }, + "panelIndex": "b3e1e89a-a86d-40b3-97e2-ba89751330fa", + "type": "lens" + }, + { + "embeddableConfig": { + "attributes": { + "references": [], + "state": { + "datasourceStates": { + "textBased": { + "indexPatternRefs": [], + "layers": { + "d720a72f-fe61-4baf-bdee-8a50d4247670": { + "columns": [ + { + "columnId": "% Normalized cost", + "customLabel": false, + "fieldName": "% Normalized cost", + "inMetricDimension": true, + "label": "% Normalized cost", + "meta": { + "esType": "double", + "type": "number" + } }, - "description": "", - "id": "", - "params": { - "fontSize": 12, - "markdown": "## Blended details", - "openLinksInNewTab": false + { + "columnId": "@timestamp", + "customLabel": false, + "fieldName": "@timestamp", + "label": "@timestamp", + "meta": { + "esType": "date", + "type": "date" + } }, - "title": "", - "type": "markdown", - "uiState": {} + { + "columnId": "datastream", + "customLabel": false, + "fieldName": "datastream", + "label": "datastream", + "meta": { + "esType": "keyword", + "type": "string" + } + } + ], + "index": "chargeback_integration", + "query": { + "esql": "FROM billing_realized_pool_lookup\n| LOOKUP JOIN cluster_capacity_utilization_lookup ON composite_key\n| LOOKUP JOIN cluster_deployment_contribution_lookup ON composite_key\n| LOOKUP JOIN cluster_tier_and_datastream_contribution_lookup ON composite_key\n| LOOKUP JOIN chargeback_conf_lookup ON @timestamp >= conf_start_date AND @timestamp <= conf_end_date\n| EVAL heap_util = COALESCE(heap_used_pct_p95, 100) / 100, disk_util = COALESCE(disk_used_pct_p95, 100) / 100, w_sum = conf_utilization_memory_weight + conf_utilization_storage_weight, capacity_utilization_score = CASE(w_sum > 0, (conf_utilization_memory_weight * heap_util + conf_utilization_storage_weight * disk_util) / w_sum, 1.0)\n| EVAL chargeable_pool = data_tier_capacity_ecu * COALESCE(capacity_utilization_score, 1.0), utilization_data_missing = heap_used_pct_p95 IS NULL AND disk_used_pct_p95 IS NULL\n| EVAL\n indexing = CASE (deployment_sum_indexing_time > 0, TO_DOUBLE(tier_and_datastream_sum_indexing_time) / deployment_sum_indexing_time * chargeable_pool) * COALESCE(conf_chargeable_unit_rate, conf_ecu_rate),\n querying = CASE (deployment_sum_query_time > 0, TO_DOUBLE(tier_and_datastream_sum_query_time) / deployment_sum_query_time * chargeable_pool) * COALESCE(conf_chargeable_unit_rate, conf_ecu_rate),\n data_set = CASE (deployment_sum_data_set_store_size > 0, TO_DOUBLE(tier_and_datastream_sum_data_set_store_size) / deployment_sum_data_set_store_size * chargeable_pool),\n store = CASE (deployment_sum_store_size > 0, TO_DOUBLE(tier_and_datastream_sum_store_size) / deployment_sum_store_size * chargeable_pool),\n storage = CASE (store == 0, data_set, store) * COALESCE(conf_chargeable_unit_rate, conf_ecu_rate),\n total_weight_hot = conf_storage_weight + conf_query_weight + conf_indexing_weight,\n total_weight_cold = conf_storage_weight + conf_query_weight,\n blended = CASE (\n tier == \"hot/content\",\n ((storage / COALESCE(conf_chargeable_unit_rate, conf_ecu_rate) * conf_storage_weight) + (querying / COALESCE(conf_chargeable_unit_rate, conf_ecu_rate) * conf_query_weight) + (indexing / COALESCE(conf_chargeable_unit_rate, conf_ecu_rate) * conf_indexing_weight)) / total_weight_hot,\n ((storage / COALESCE(conf_chargeable_unit_rate, conf_ecu_rate) * conf_storage_weight) + (querying / COALESCE(conf_chargeable_unit_rate, conf_ecu_rate) * conf_query_weight)) / total_weight_cold\n ) * COALESCE(conf_chargeable_unit_rate, conf_ecu_rate)\n| STATS\n `% Normalized cost` = sum(blended)\n BY\n @timestamp,\n datastream,\n tier\n| WHERE `% Normalized cost` > 0\n| SORT `% Normalized cost` DESC" + }, + "timeField": "@timestamp" } - }, - "gridData": { - "h": 3, - "i": "50e188d3-b5da-49e7-8342-247554634e9a", - "sectionId": "7f7cc9b5-13b3-4962-98f6-5b18a1d8ca80", - "w": 48, - "x": 0, - "y": 55 - }, - "panelIndex": "50e188d3-b5da-49e7-8342-247554634e9a", - "type": "visualization" + } + } + }, + "filters": [], + "needsRefresh": false, + "query": { + "esql": "FROM billing_realized_pool_lookup\n| LOOKUP JOIN cluster_capacity_utilization_lookup ON composite_key\n| LOOKUP JOIN cluster_deployment_contribution_lookup ON composite_key\n| LOOKUP JOIN cluster_tier_and_datastream_contribution_lookup ON composite_key\n| LOOKUP JOIN chargeback_conf_lookup ON @timestamp >= conf_start_date AND @timestamp <= conf_end_date\n| EVAL heap_util = COALESCE(heap_used_pct_p95, 100) / 100, disk_util = COALESCE(disk_used_pct_p95, 100) / 100, w_sum = conf_utilization_memory_weight + conf_utilization_storage_weight, capacity_utilization_score = CASE(w_sum > 0, (conf_utilization_memory_weight * heap_util + conf_utilization_storage_weight * disk_util) / w_sum, 1.0)\n| EVAL chargeable_pool = data_tier_capacity_ecu * COALESCE(capacity_utilization_score, 1.0), utilization_data_missing = heap_used_pct_p95 IS NULL AND disk_used_pct_p95 IS NULL\n| EVAL\n indexing = CASE (deployment_sum_indexing_time > 0, TO_DOUBLE(tier_and_datastream_sum_indexing_time) / deployment_sum_indexing_time * chargeable_pool) * COALESCE(conf_chargeable_unit_rate, conf_ecu_rate),\n querying = CASE (deployment_sum_query_time > 0, TO_DOUBLE(tier_and_datastream_sum_query_time) / deployment_sum_query_time * chargeable_pool) * COALESCE(conf_chargeable_unit_rate, conf_ecu_rate),\n data_set = CASE (deployment_sum_data_set_store_size > 0, TO_DOUBLE(tier_and_datastream_sum_data_set_store_size) / deployment_sum_data_set_store_size * chargeable_pool),\n store = CASE (deployment_sum_store_size > 0, TO_DOUBLE(tier_and_datastream_sum_store_size) / deployment_sum_store_size * chargeable_pool),\n storage = CASE (store == 0, data_set, store) * COALESCE(conf_chargeable_unit_rate, conf_ecu_rate),\n total_weight_hot = conf_storage_weight + conf_query_weight + conf_indexing_weight,\n total_weight_cold = conf_storage_weight + conf_query_weight,\n blended = CASE (\n tier == \"hot/content\",\n ((storage / COALESCE(conf_chargeable_unit_rate, conf_ecu_rate) * conf_storage_weight) + (querying / COALESCE(conf_chargeable_unit_rate, conf_ecu_rate) * conf_query_weight) + (indexing / COALESCE(conf_chargeable_unit_rate, conf_ecu_rate) * conf_indexing_weight)) / total_weight_hot,\n ((storage / COALESCE(conf_chargeable_unit_rate, conf_ecu_rate) * conf_storage_weight) + (querying / COALESCE(conf_chargeable_unit_rate, conf_ecu_rate) * conf_query_weight)) / total_weight_cold\n ) * COALESCE(conf_chargeable_unit_rate, conf_ecu_rate)\n| STATS\n `% Normalized cost` = sum(blended)\n BY\n @timestamp,\n datastream,\n tier\n| WHERE `% Normalized cost` > 0\n| SORT `% Normalized cost` DESC" + }, + "visualization": { + "axisTitlesVisibilitySettings": { + "x": true, + "yLeft": true, + "yRight": true + }, + "fittingFunction": "Linear", + "gridlinesVisibilitySettings": { + "x": true, + "yLeft": true, + "yRight": true + }, + "labelsOrientation": { + "x": 0, + "yLeft": 0, + "yRight": 0 + }, + "layers": [ + { + "accessors": [ + "% Normalized cost" + ], + "colorMapping": { + "assignments": [], + "colorMode": { + "type": "categorical" + }, + "paletteId": "default", + "specialAssignments": [ + { + "color": { + "type": "loop" + }, + "rules": [ + { + "type": "other" + } + ], + "touched": false + } + ] + }, + "layerId": "d720a72f-fe61-4baf-bdee-8a50d4247670", + "layerType": "data", + "seriesType": "bar_percentage_stacked", + "splitAccessor": "datastream", + "splitAccessors": [ + "datastream" + ], + "xAccessor": "@timestamp" + } + ], + "legend": { + "isVisible": false, + "position": "right", + "showSingleSeries": false + }, + "preferredSeriesType": "bar", + "tickLabelsVisibilitySettings": { + "x": true, + "yLeft": true, + "yRight": true + }, + "valueLabels": "hide" + } + }, + "title": "Blended Cost Share by Tier and Data Stream (%)", + "version": 2, + "visualizationType": "lnsXY" + }, + "title": "Blended Cost Share by Tier and Data Stream (%)" + }, + "gridData": { + "h": 14, + "i": "cefe7754-4e91-4b85-8132-512a9ca31793", + "sectionId": "7f7cc9b5-13b3-4962-98f6-5b18a1d8ca80", + "w": 16, + "x": 32, + "y": 58 + }, + "panelIndex": "cefe7754-4e91-4b85-8132-512a9ca31793", + "type": "lens" + }, + { + "embeddableConfig": { + "enhancements": { + "dynamicActions": { + "events": [] } - ], - "refreshInterval": { - "pause": true, - "value": 60000 + }, + "savedVis": { + "data": { + "aggs": [], + "searchSource": { + "filter": [], + "query": { + "language": "kuery", + "query": "" + } + } + }, + "description": "", + "id": "", + "params": { + "fontSize": 12, + "markdown": "# Data tier and data stream per day", + "openLinksInNewTab": false + }, + "title": "", + "type": "markdown", + "uiState": {} + } }, - "sections": [ - { - "collapsed": false, - "gridData": { - "i": "0fceb22c-1e73-45e4-b634-006f6dcc5f43", - "y": 2 - }, - "title": "Datatiers / utilization" + "gridData": { + "h": 3, + "i": "5994f908-7b57-4bd9-8e58-0b275c4c7f8e", + "sectionId": "7f7cc9b5-13b3-4962-98f6-5b18a1d8ca80", + "w": 48, + "x": 0, + "y": 0 + }, + "panelIndex": "5994f908-7b57-4bd9-8e58-0b275c4c7f8e", + "type": "visualization" + }, + { + "embeddableConfig": { + "enhancements": { + "dynamicActions": { + "events": [] + } + }, + "savedVis": { + "data": { + "aggs": [], + "searchSource": { + "filter": [], + "query": { + "language": "kuery", + "query": "" + } + } }, - { - "collapsed": false, - "gridData": { - "i": "d11ae80b-d7bb-474f-9082-81752ab2865d", - "y": 3 - }, - "title": "Data tier and data stream overview" + "description": "", + "id": "", + "params": { + "fontSize": 12, + "markdown": "## Indexing details", + "openLinksInNewTab": false }, - { - "collapsed": false, - "gridData": { - "i": "7f7cc9b5-13b3-4962-98f6-5b18a1d8ca80", - "y": 4 - }, - "title": "Data tier and data stream per day" + "title": "", + "type": "markdown", + "uiState": {} + } + }, + "gridData": { + "h": 3, + "i": "5011c332-ce8a-4a78-a08a-c0527c374fcb", + "sectionId": "7f7cc9b5-13b3-4962-98f6-5b18a1d8ca80", + "w": 48, + "x": 0, + "y": 3 + }, + "panelIndex": "5011c332-ce8a-4a78-a08a-c0527c374fcb", + "type": "visualization" + }, + { + "embeddableConfig": { + "enhancements": { + "dynamicActions": { + "events": [] } - ], - "timeFrom": "now-3M", - "timeRestore": true, - "timeTo": "now", - "title": "[Chargeback] Usage & Cost Allocation", - "version": 1 + }, + "savedVis": { + "data": { + "aggs": [], + "searchSource": { + "filter": [], + "query": { + "language": "kuery", + "query": "" + } + } + }, + "description": "", + "id": "", + "params": { + "fontSize": 12, + "markdown": "## Querying details", + "openLinksInNewTab": false + }, + "title": "", + "type": "markdown", + "uiState": {} + } + }, + "gridData": { + "h": 3, + "i": "e458e034-197a-4ec6-95a8-2938181a4883", + "sectionId": "7f7cc9b5-13b3-4962-98f6-5b18a1d8ca80", + "w": 48, + "x": 0, + "y": 20 + }, + "panelIndex": "e458e034-197a-4ec6-95a8-2938181a4883", + "type": "visualization" + }, + { + "embeddableConfig": { + "enhancements": { + "dynamicActions": { + "events": [] + } + }, + "savedVis": { + "data": { + "aggs": [], + "searchSource": { + "filter": [], + "query": { + "language": "kuery", + "query": "" + } + } + }, + "description": "", + "id": "", + "params": { + "fontSize": 12, + "markdown": "## Storage details", + "openLinksInNewTab": false + }, + "title": "", + "type": "markdown", + "uiState": {} + } + }, + "gridData": { + "h": 3, + "i": "692b3cdd-b37f-4da9-80a6-aff7b799ecc1", + "sectionId": "7f7cc9b5-13b3-4962-98f6-5b18a1d8ca80", + "w": 48, + "x": 0, + "y": 37 + }, + "panelIndex": "692b3cdd-b37f-4da9-80a6-aff7b799ecc1", + "type": "visualization" + }, + { + "embeddableConfig": { + "enhancements": { + "dynamicActions": { + "events": [] + } + }, + "savedVis": { + "data": { + "aggs": [], + "searchSource": { + "filter": [], + "query": { + "language": "kuery", + "query": "" + } + } + }, + "description": "", + "id": "", + "params": { + "fontSize": 12, + "markdown": "## Blended details", + "openLinksInNewTab": false + }, + "title": "", + "type": "markdown", + "uiState": {} + } + }, + "gridData": { + "h": 3, + "i": "50e188d3-b5da-49e7-8342-247554634e9a", + "sectionId": "7f7cc9b5-13b3-4962-98f6-5b18a1d8ca80", + "w": 48, + "x": 0, + "y": 55 + }, + "panelIndex": "50e188d3-b5da-49e7-8342-247554634e9a", + "type": "visualization" + } + ], + "refreshInterval": { + "pause": true, + "value": 60000 }, - "id": "chargeback-21588d0e-fb6a-4f76-ad2f-cd7b3d7a3d7c", - "references": [ - { - "id": "chargeback_integration", - "name": "controlGroup_b680ef70-4697-41e5-bf5f-e7e1e700ec3e:optionsListDataView", - "type": "index-pattern" - }, - { - "id": "chargeback_integration", - "name": "controlGroup_2c7df183-8a05-4ee0-b7db-415e874636cd:optionsListDataView", - "type": "index-pattern" - }, - { - "id": "chargeback-b6cecac0-ebe7-43de-bfc6-1ff287ad860a", - "name": "13cd2a6a-ec89-4553-ae64-cbc347770aa5:link_ea0417fa-630d-c656-8f4b-a8d7b9916b29_dashboard", - "type": "dashboard" - }, - { - "id": "chargeback-21588d0e-fb6a-4f76-ad2f-cd7b3d7a3d7c", - "name": "13cd2a6a-ec89-4553-ae64-cbc347770aa5:link_dc802531-4211-c74b-d9db-b197b528bbc8_dashboard", - "type": "dashboard" - }, - { - "id": "chargeback-7ec9743a-ccfa-43d2-8311-2f25893dadae", - "name": "13cd2a6a-ec89-4553-ae64-cbc347770aa5:link_4e06e36e-f5d7-981f-8554-57756d1c59b8_dashboard", - "type": "dashboard" - } + "sections": [ + { + "collapsed": false, + "gridData": { + "i": "0fceb22c-1e73-45e4-b634-006f6dcc5f43", + "y": 2 + }, + "title": "Datatiers / utilization" + }, + { + "collapsed": false, + "gridData": { + "i": "d11ae80b-d7bb-474f-9082-81752ab2865d", + "y": 3 + }, + "title": "Data tier and data stream overview" + }, + { + "collapsed": false, + "gridData": { + "i": "7f7cc9b5-13b3-4962-98f6-5b18a1d8ca80", + "y": 4 + }, + "title": "Data tier and data stream per day" + } ], - "type": "dashboard", - "typeMigrationVersion": "10.3.0", - "coreMigrationVersion": "8.8.0" -} \ No newline at end of file + "timeFrom": "now-3M", + "timeRestore": true, + "timeTo": "now", + "title": "[Chargeback] Usage & Cost Allocation", + "version": 1 + }, + "id": "chargeback-21588d0e-fb6a-4f76-ad2f-cd7b3d7a3d7c", + "references": [ + { + "id": "chargeback_integration", + "name": "controlGroup_b680ef70-4697-41e5-bf5f-e7e1e700ec3e:optionsListDataView", + "type": "index-pattern" + }, + { + "id": "chargeback_integration", + "name": "controlGroup_2c7df183-8a05-4ee0-b7db-415e874636cd:optionsListDataView", + "type": "index-pattern" + }, + { + "id": "chargeback-b6cecac0-ebe7-43de-bfc6-1ff287ad860a", + "name": "13cd2a6a-ec89-4553-ae64-cbc347770aa5:link_ea0417fa-630d-c656-8f4b-a8d7b9916b29_dashboard", + "type": "dashboard" + }, + { + "id": "chargeback-21588d0e-fb6a-4f76-ad2f-cd7b3d7a3d7c", + "name": "13cd2a6a-ec89-4553-ae64-cbc347770aa5:link_dc802531-4211-c74b-d9db-b197b528bbc8_dashboard", + "type": "dashboard" + }, + { + "id": "chargeback-7ec9743a-ccfa-43d2-8311-2f25893dadae", + "name": "13cd2a6a-ec89-4553-ae64-cbc347770aa5:link_4e06e36e-f5d7-981f-8554-57756d1c59b8_dashboard", + "type": "dashboard" + } + ], + "type": "dashboard", + "typeMigrationVersion": "10.3.0", + "coreMigrationVersion": "8.8.0" +} diff --git a/packages/chargeback/manifest.yml b/packages/chargeback/manifest.yml index e6e2228bc4d..c91492d3dbb 100644 --- a/packages/chargeback/manifest.yml +++ b/packages/chargeback/manifest.yml @@ -1,7 +1,7 @@ format_version: 3.5.0 name: chargeback title: "Chargeback" -version: 0.4.0 +version: 0.4.1 description: "This package calculates chargeback based on billing and consumption data" type: integration categories: From e12347c5902a5026cfbcf298503b99d0382287a0 Mon Sep 17 00:00:00 2001 From: Johannes Mahne Date: Mon, 3 Aug 2026 17:29:49 +0200 Subject: [PATCH 2/3] [Chargeback] Fix empty Usage dashboard: sync billing transforms on @timestamp billing_realized_pool and billing_cluster_cost synced on event.ingested, which is often unmapped on On-Prem Billing indices, leaving realized_pool empty and Usage panels blank. Co-authored-by: Cursor --- packages/chargeback/changelog.yml | 3 +++ .../transform/billing_cluster_cost/transform.yml | 4 +++- .../transform/billing_realized_pool/transform.yml | 5 ++++- 3 files changed, 10 insertions(+), 2 deletions(-) diff --git a/packages/chargeback/changelog.yml b/packages/chargeback/changelog.yml index ad6e949a452..ff43c8fe0bf 100644 --- a/packages/chargeback/changelog.yml +++ b/packages/chargeback/changelog.yml @@ -13,6 +13,9 @@ - description: "Document expected small deltas versus the ESS Billing dashboard for incomplete UTC days and transform sync delay; compare completed days when reconciling." type: enhancement link: https://github.com/elastic/elasticsearch-chargeback/issues/66 + - description: "Fix empty Usage and Cost Allocation dashboard when billing sources lack a mapped event.ingested field (common with On-Prem Billing). billing_cluster_cost and billing_realized_pool now sync on @timestamp so billing_realized_pool_lookup populates and usage panels can render." + type: bugfix + link: https://github.com/elastic/elasticsearch-chargeback/issues/23 - version: 0.4.0 changes: - description: "Split monolithic dashboard into three focused dashboards: '[Chargeback] Billing Components Overview' (full invoice by deployment group and billing component), '[Chargeback] Usage & Cost Allocation' (usage-based chargeable pool allocation by tier and data stream), and '[Chargeback] Configuration' (rate, weights, and date-window reference). Resolves elasticsearch-chargeback#8." diff --git a/packages/chargeback/elasticsearch/transform/billing_cluster_cost/transform.yml b/packages/chargeback/elasticsearch/transform/billing_cluster_cost/transform.yml index 57a472d4922..4214ab21fb9 100644 --- a/packages/chargeback/elasticsearch/transform/billing_cluster_cost/transform.yml +++ b/packages/chargeback/elasticsearch/transform/billing_cluster_cost/transform.yml @@ -34,7 +34,9 @@ dest: frequency: 60m sync: time: - field: event.ingested + # Align with billing_realized_pool: @timestamp is always mapped on billing sources. + # event.ingested is often unset/unmapped on On-Prem Billing indices. + field: "@timestamp" delay: 1h pivot: group_by: diff --git a/packages/chargeback/elasticsearch/transform/billing_realized_pool/transform.yml b/packages/chargeback/elasticsearch/transform/billing_realized_pool/transform.yml index bb61fc8dab9..9f670ba96d1 100644 --- a/packages/chargeback/elasticsearch/transform/billing_realized_pool/transform.yml +++ b/packages/chargeback/elasticsearch/transform/billing_realized_pool/transform.yml @@ -59,7 +59,10 @@ dest: frequency: 60m sync: time: - field: event.ingested + # Use @timestamp (billing day), not event.ingested. On-Prem Billing and some + # seeded sources do not map event.ingested, which left this transform at 0 docs + # and emptied the Usage & Cost Allocation dashboard. + field: "@timestamp" delay: 1h pivot: group_by: From 1dc11f9bc699025cd861cf07f811eacbc38d3cd0 Mon Sep 17 00:00:00 2001 From: Johannes Mahne Date: Mon, 3 Aug 2026 17:30:19 +0200 Subject: [PATCH 3/3] [Chargeback] Document realized_pool sync and empty Usage dashboard check Co-authored-by: Cursor --- packages/chargeback/_dev/build/docs/README.md | 2 ++ packages/chargeback/docs/README.md | 2 ++ 2 files changed, 4 insertions(+) diff --git a/packages/chargeback/_dev/build/docs/README.md b/packages/chargeback/_dev/build/docs/README.md index ae210517652..39d7853561e 100755 --- a/packages/chargeback/_dev/build/docs/README.md +++ b/packages/chargeback/_dev/build/docs/README.md @@ -39,6 +39,8 @@ Chargeback costs are presented based on a configured rate and unit, used to conv 2. Install and start the **Elasticsearch** integration (index pivot + node stats) on monitored deployments. 3. Install **Chargeback**. The `chargeback_conf_lookup` bootstrap transform uses the billing metrics pattern as its source trigger. If Chargeback starts before any billing backing index exists, that transform can fail to start until billing data appears; start or `_schedule_now` the transform after billing indices exist. +Billing cost transforms (`billing_cluster_cost`, `billing_realized_pool`) sync on `@timestamp`. If the Usage and Cost Allocation dashboard is empty, confirm `billing_realized_pool_lookup` has documents — that index is the primary source for usage panels. + **Data flow:** 1. ESS Billing data is collected into `metrics-ess_billing.billing-*`. 2. Elasticsearch index usage data is aggregated into `monitoring-indices*` by the Elasticsearch integration index pivot transform. diff --git a/packages/chargeback/docs/README.md b/packages/chargeback/docs/README.md index ae210517652..39d7853561e 100644 --- a/packages/chargeback/docs/README.md +++ b/packages/chargeback/docs/README.md @@ -39,6 +39,8 @@ Chargeback costs are presented based on a configured rate and unit, used to conv 2. Install and start the **Elasticsearch** integration (index pivot + node stats) on monitored deployments. 3. Install **Chargeback**. The `chargeback_conf_lookup` bootstrap transform uses the billing metrics pattern as its source trigger. If Chargeback starts before any billing backing index exists, that transform can fail to start until billing data appears; start or `_schedule_now` the transform after billing indices exist. +Billing cost transforms (`billing_cluster_cost`, `billing_realized_pool`) sync on `@timestamp`. If the Usage and Cost Allocation dashboard is empty, confirm `billing_realized_pool_lookup` has documents — that index is the primary source for usage panels. + **Data flow:** 1. ESS Billing data is collected into `metrics-ess_billing.billing-*`. 2. Elasticsearch index usage data is aggregated into `monitoring-indices*` by the Elasticsearch integration index pivot transform.