diff --git a/CHANGELOG.md b/CHANGELOG.md index c6bfedd..770a937 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -10,6 +10,21 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/). ## Integration Releases +### [0.5.0] - 2026-06-05 + +#### Changed + +- Billing Components Overview and Usage & Cost Allocation dashboards: legacy options-list controls replaced with chained ES|QL multi-select variable controls (`MV_CONTAINS` filtering). +- Kibana minimum version: **9.4.0+** (Elasticsearch remains **9.2.0+**). **Kibana 9.3 is not supported** — Billing and Usage dashboards fail with `No embeddable factory found for type: vis`. Transform `fleet_transform_version` and pipeline references bumped to `0.5.0` (pipeline logic unchanged). + +#### Upgrade notes + +When upgrading from 0.4.x: +1. Upgrade Kibana to **9.4.0+** before installing 0.5.0 (9.3 is unsupported). +2. Upgrade the Fleet package to **0.5.0**; replace duplicate Billing/Usage dashboard saved objects if needed. + +--- + ### [0.4.0] - 2026-06-01 #### Added diff --git a/integration/Instructions.md b/integration/Instructions.md index 9c9bc2c..92d3c32 100644 --- a/integration/Instructions.md +++ b/integration/Instructions.md @@ -14,12 +14,14 @@ To install the Chargeback integration, please follow these steps: See [Requirements](README.md#requirements) for details. -**Key prerequisite for 0.4.0:** The [Elasticsearch integration](https://www.elastic.co/docs/reference/integrations/elasticsearch/) must be **actively running** on all monitored deployments with both **index stats** and **node stats** datasets enabled. Node stats are required for the realized cost utilization score. Without them, utilization defaults to 100% and no discount is applied. +**Key prerequisite for 0.4.0+:** The [Elasticsearch integration](https://www.elastic.co/docs/reference/integrations/elasticsearch/) must be **actively running** on all monitored deployments with both **index stats** and **node stats** datasets enabled. Node stats are required for the realized cost utilization score. Without them, utilization defaults to 100% and no discount is applied. + +**Key prerequisite for 0.5.0+:** Kibana **9.4.0+**. **Kibana 9.3 is not supported** — Billing and Usage dashboards fail to render with `No embeddable factory found for type: vis` and do not support GA ES|QL multi-select variable controls. ### 2. Upload ZIP File: -- Asset: [`chargeback-0.4.0.zip`](assets/0.4.0/chargeback-0.4.0.zip) -- Browse to Integrations, and click on `+ Create new integration` +- Asset: [`chargeback-0.5.1.zip`](assets/0.5.1/chargeback-0.5.1.zip) +- Browse to Integrations, and select **+ Create new integration** ![alt text](assets/img/CreateNewIntegration.png) @@ -31,7 +33,7 @@ See [Requirements](README.md#requirements) for details. Starting from version 0.2.8, all Chargeback transforms are configured to auto-start upon installation. You no longer need to manually start the transforms. -**Starting from version 0.2.10** (current: **v0.4.0**), the `chargeback_conf_lookup` index is automatically created via a bootstrap transform during installation. No manual setup is required! The transform creates the index with default configuration: +**Starting from version 0.2.10** (current: **v0.5.1**), the `chargeback_conf_lookup` index is automatically created via a bootstrap transform during installation. No manual setup is required! The transform creates the index with default configuration: - **Chargeable unit rate:** 0.85 EUR - **Weights:** indexing=20, query=20, storage=40 - **Date range:** 2010-01-01 to 2046-12-31 @@ -57,6 +59,16 @@ To upgrade the integration, do the following: - Upload the new asset (ZIP) file to Kibana. - Transforms will auto-start (from version 0.2.8 onwards). +**Upgrading from 0.5.0 to 0.5.1:** +- Upload `chargeback-0.5.1.zip` (Kibana **9.4.0+**). +- Usage transforms reinstall with `fleet_transform_version: 0.5.1` and add `ds_type` / `ds_namespace`. Reset those transforms if existing lookup docs lack the new fields. +- Usage & Cost Allocation gains data stream type and namespace controls. Delete duplicate dashboard saved objects if they appear after upgrade. + +**Upgrading from 0.4.x to 0.5.0:** +- Upgrade Kibana to **9.4.0+**, then upload `chargeback-0.5.0.zip`. Do not install on Kibana 9.3. +- Transforms are reinstalled with `fleet_transform_version: 0.5.0` (pipeline logic unchanged). No lookup index recreation is required. +- Billing and Usage dashboards are replaced with ES|QL multi-select variable controls. Delete duplicate dashboard saved objects if they appear after upgrade. + **Upgrading from 0.3.x to 0.4.0:** - Upload `chargeback-0.4.0.zip`. Two new transforms (`billing_realized_pool`, `cluster_capacity_utilization`) are created and auto-started. - The old `[Chargeback] Cost and Consumption breakdown` dashboard is replaced by three new dashboards (`[Chargeback] Billing Components Overview`, `[Chargeback] Usage & Cost Allocation`, and `[Chargeback] Configuration`). If Kibana does not replace the old dashboard automatically, delete it and re-import from **Saved Objects**. @@ -81,7 +93,7 @@ If the dashboard is empty after install, see **[docs/troubleshooting.md](docs/tr ## Configuration -Configuration values are stored in the `chargeback_conf_lookup` index, which is automatically created by version 0.2.10+ (current: **v0.4.0**). The dashboard automatically applies the correct configuration based on the billing date falling within the `conf_start_date` and `conf_end_date` range. +Configuration values are stored in the `chargeback_conf_lookup` index, which is automatically created by version 0.2.10+ (current: **v0.5.0**). The dashboard automatically applies the correct configuration based on the billing date falling within the `conf_start_date` and `conf_end_date` range. ### Update the default configuration: diff --git a/integration/README.md b/integration/README.md index f1b799b..45f5f8c 100644 --- a/integration/README.md +++ b/integration/README.md @@ -2,7 +2,7 @@ ## Version -Chargeback integration: 0.4.0 +Chargeback integration: 0.5.1 ## Dependencies @@ -12,7 +12,8 @@ This process must be set up on the **Monitoring cluster**, where all monitoring To use this integration, the following prerequisites must be met: -- The monitoring cluster, where this integration is installed, must be on version 9.2.0+ due to its use of (smart) [ES|QL LOOKUP JOIN](https://www.elastic.co/docs/reference/query-languages/esql/esql-lookup-join). +- The monitoring cluster, where this integration is installed, must be on Elasticsearch version 9.2.0+ due to its use of (smart) [ES|QL LOOKUP JOIN](https://www.elastic.co/docs/reference/query-languages/esql/esql-lookup-join). +- Kibana version **9.4.0+** is required for integration **0.5.0** and later. **Kibana 9.3 is not supported** — Billing and Usage dashboards fail to render (`No embeddable factory found for type: vis`) and lack GA support for ES|QL multi-select variable controls. - The [**Elasticsearch Service Billing**](https://www.elastic.co/docs/reference/integrations/ess_billing/) integration (v1.7.0+) must be installed and running. - The [**Elasticsearch**](https://www.elastic.co/docs/reference/integrations/elasticsearch/) integration (v1.16.0+) must be **installed and actively running** on all monitored deployments, with the following datasets enabled: - **Index stats** — required for tier and data stream cost allocation. The `logs-elasticsearch.index_pivot-default-{VERSION}` transform must be running to aggregate these into `monitoring-indices`. @@ -29,7 +30,9 @@ This integration must be installed on the **Monitoring cluster** where the above | 0.2.10 - 0.2.x | 9.2.0+ | 1.7.0+ | Requires ESS Billing 1.7.0 features | | 0.3.0 | 9.2.0+ | 1.7.0+ | Chargeable units schema (breaking change from 0.2.x) | | 0.3.1 - 0.3.2 | 9.2.0+ | 1.7.0+ | Field renames, deployment_tags fix, explicit lookup mappings | -| 0.4.0+ | 9.2.0+ | 1.7.0+ | Realized cost model, SKU classification, three-dashboard split | +| 0.4.0 | 9.2.0+ | 1.7.0+ | Realized cost model, SKU classification, three-dashboard split | +| 0.5.0+ | 9.4.0+ (Kibana), 9.2.0+ (ES). **Not 9.3** | 1.7.0+ | ES|QL multi-select variable controls; Kibana 9.3 fails with `No embeddable factory found for type: vis` | +| 0.5.1 | 9.4.0+ (Kibana), 9.2.0+ (ES). **Not 9.3** | 1.7.0+ | `ds_type` / `ds_namespace` on usage path; Usage dashboard type and namespace controls | ## Setup instructions @@ -58,8 +61,8 @@ The first layer of processing that we do, is eight transforms: **Usage transforms** (from monitoring indices): - `cluster_deployment_contribution` — indexing, querying, and storage metrics per deployment per day. - `cluster_tier_contribution` — same metrics split by data tier. -- `cluster_datastream_contribution` — same metrics split by data stream. -- `cluster_tier_and_ds_contribution` — same metrics split by both tier and data stream. +- `cluster_datastream_contribution` — same metrics split by data stream; usage pipeline sets `ds_type` and `ds_namespace`. +- `cluster_tier_and_ds_contribution` — same metrics split by both tier and data stream (includes `ds_type` and `ds_namespace`). ![Transforms](assets/img/Transforms.png) @@ -94,10 +97,16 @@ Answers: *what did we spend and where did it go?* Answers: *which data streams and tiers drive cost, and how efficiently are we using capacity?* +Controls include deployment group, deployment name, data tier, full data stream name, **data stream type** (`ds_type`), and **data stream namespace** (`ds_namespace`). + - **Data tiers / utilization** — provisioned capacity versus realized pool (`chargeable_pool = provisioned × util_score`), p95 heap and disk utilization. - **Data tier and data stream overview** — top-20 data streams by indexing / query / storage cost, blended cost totals, workload mix by tier. - **Data tier and data stream per day** — time-series cost breakdown (indexing, querying, storage, blended) by data stream and tier, including percentage share panels. +### Shared deployments and namespaces + +On a shared deployment, assign each team a unique [Fleet data stream namespace](https://www.elastic.co/docs/reference/fleet/data-streams) (`type-dataset-namespace`). Chargeback parses `datastream` into `ds_type` (first segment) and `ds_namespace` (last segment), with `other` when the name has no hyphen. Use the Usage dashboard type and namespace controls for team-level FinOps. A dataset control is not included in 0.5.1. + ### [Chargeback] Configuration A standalone reference dashboard showing all active configuration values: conversion rate, date windows, blended cost weights, utilization score weights, and memory/storage cost split — each visualised as a percentage-stacked bar chart. @@ -114,6 +123,32 @@ These alerting templates are automatically installed with the integration and ca **Important:** For alert rules 2 and 3, ensure that the Chargeback transforms are running before setting them up. These alerting rules query the lookup indices created by the transforms (`billing_cluster_cost_lookup`, `cluster_deployment_contribution_lookup`, etc.). If the transforms are not started, the alerts will not function correctly. +## Version 0.5.1 Release Notes + +### Added + +- Usage ingest pipeline materializes `ds_type` and `ds_namespace` from each `datastream` name (Fleet `type-dataset-namespace` scheme; `other` fallback). +- Usage & Cost Allocation dashboard: **Data stream type** and **Data stream namespace** ES|QL multi-select controls. +- Docs for namespace-based ownership on shared deployments. + +### Upgrade from 0.5.0 + +1. Upload `chargeback-0.5.1.zip` (Kibana **9.4.0+**). +2. Reset usage transforms that write through the usage pipeline if existing lookup docs lack `ds_type` / `ds_namespace`. +3. Replace Usage dashboard saved objects if duplicates appear after upgrade. + +## Version 0.5.0 Release Notes + +### Changed + +- **Billing and Usage dashboards** use chained ES|QL multi-select variable controls instead of legacy options-list controls. Panel queries filter with `MV_INTERSECTS` and a `__chargeback_unfiltered__` sentinel default. +- Package `kibana.version` constraint raised to `^9.4.0`. **Kibana 9.3 is not supported** — dashboards fail with `No embeddable factory found for type: vis`. Transform `fleet_transform_version` and pipeline references bumped to `0.5.0` (pipeline logic unchanged). + +### Upgrade from 0.4.x + +1. Upgrade Kibana to **9.4.0+**, then upgrade the Fleet package to **0.5.0**. Do not install 0.5.0 on Kibana 9.3. +2. Re-import or replace the Billing and Usage dashboard saved objects if duplicates appear after upgrade. + ## Version 0.4.0 Release Notes ### Added diff --git a/integration/assets/0.5.0/chargeback-0.5.0.zip b/integration/assets/0.5.0/chargeback-0.5.0.zip new file mode 100644 index 0000000..5ef8c6d Binary files /dev/null and b/integration/assets/0.5.0/chargeback-0.5.0.zip differ diff --git a/integration/assets/0.5.1/chargeback-0.5.1.zip b/integration/assets/0.5.1/chargeback-0.5.1.zip new file mode 100644 index 0000000..160aa7c Binary files /dev/null and b/integration/assets/0.5.1/chargeback-0.5.1.zip differ diff --git a/integration/docs/troubleshooting.md b/integration/docs/troubleshooting.md index 8974994..29c4675 100644 --- a/integration/docs/troubleshooting.md +++ b/integration/docs/troubleshooting.md @@ -244,10 +244,89 @@ The bundled dashboard is **Fleet/Kibana managed**—you cannot fix this by editi There is no supported workaround on **0.3.1** other than upgrading the package. +## `x_content_parse_exception`: empty lists in ES|QL `params` (0.5.0 on Elasticsearch 9.4+) + +### Symptom + +Billing or Usage dashboard panels fail with: + +`Unexpected error from Elasticsearch: x_content_parse_exception - [esql/async_query] failed to parse field [params]` + +and messages like: + +`Empty lists are not allowed as named parameter values. Got parameter [dg_selected] with value [[]]` + +Kibana may also report **“N panels have been removed from the dashboard”** after a bad dashboard import or manual edit that strips ES|QL filter clauses or replaces ES|QL controls with legacy options-list controls. + +### Cause + +**0.5.0** Billing and Usage dashboards use chained **ES|QL multi-select variable controls** (`variable_type: multi_values`) with this filter pattern: + +```esql +WHERE (?dg_selected IS NULL OR MV_CONTAINS(?dg_selected, deployment_group)) +``` + +When no values are selected, Kibana sends **empty arrays** in the ES|QL request: + +```json +"params": [{ "dg_selected": [] }, { "deployment_selected": [] }] +``` + +From **Elasticsearch 9.4** onward, **named ES|QL parameters cannot be `[]`**. Elasticsearch rejects the request at parse time ([elastic/elasticsearch#147448](https://github.com/elastic/elasticsearch/issues/147448), [PR #147748](https://github.com/elastic/elasticsearch/pull/147748)). The `?var IS NULL` branch never runs because the parameter value is `[]`, not SQL `NULL`. + +The Chargeback query pattern is correct. The bug is in **Kibana param serialization**: `esql_control_manager.ts` sets `value = selectedValues.map(...)` for multi-select (which is `[]` when nothing is selected), and `getNamedParams()` in `@kbn/esql-utils` forwards that value unchanged to Elasticsearch. + +**Do not “fix” this in the Chargeback package by:** + +- Removing `?variable` filters from panel ES|QL (breaks multi-select filtering; may drop panels on import) +- Reverting to legacy `options_list_control` / `controlGroupInput` (removes chained tier / data stream / cost category / cost type filters) +- Adding an `"All"` sentinel value (breaks multi-select UX) + +### Verified behaviour on Elasticsearch 9.4 + +| Request param | Result with `?var IS NULL OR MV_CONTAINS(?var, field)` | +|---------------|--------------------------------------------------------| +| `[{"var": null}]` | Works — shows all rows (no filter) | +| `[{"var": ["product"]}]` | Works — filters correctly | +| `[{"var": []}]` | **400 parse error** (same error you see in Kibana) | +| Param omitted entirely | **Unknown query parameter** error | + +### Workaround (until Kibana is patched) + +On each ES|QL control in the Billing / Usage dashboard, open the control and choose **Select all** so `selected_options` is populated with real values instead of `[]`. Save the dashboard if you want that state to persist. + +This is a UX workaround only. Fresh installs of **0.5.0** still ship with `"selected_options": []` on each control. + +### Upstream fix (Kibana) + +Kibana should send `null` (one param object per variable) when a `multi_values` control has no selection, matching the Agent Builder pattern in [kibana#256588](https://github.com/elastic/kibana/pull/256588). Minimal fix in `getNamedParams()` (`src/platform/packages/shared/kbn-esql-utils/src/utils/run_query.ts`): + +```typescript +if (type === ESQLVariableType.MULTI_VALUES && Array.isArray(value) && value.length === 0) { + namedParams.push({ [key]: null }); +} else { + namedParams.push({ [key]: value }); +} +``` + +See also `scripts/DESIGN_ESQL_EMPTY_PARAMS.md` in this repository for full analysis. + +### If panels were removed from the dashboard + +Reinstall **0.5.0** dashboards from the integration package (do not hand-edit controls back to options-list). With `REPLACE_CHARGEBACK_DASHBOARD=1`: + +```bash +REPLACE_CHARGEBACK_DASHBOARD=1 ./scripts/run_e2e_tests.sh +``` + +Or delete the Billing / Usage dashboard saved objects in **Stack Management → Saved Objects** and reinstall the Chargeback integration zip. + ## Symptom → likely cause | Symptom | Likely cause | |---------|----------------| +| `Empty lists are not allowed as named parameter values` on Billing/Usage (0.5.0, ES 9.4+) | Kibana sends `[]` for empty multi-select ES|QL controls; needs Kibana fix or **Select all** workaround | +| “N panels have been removed from the dashboard” after upgrade | Corrupt dashboard import (stripped ES|QL filters or wrong control type); reinstall 0.5.0 dashboards from package | | All panels empty; Chargeback transforms “started” | `logs-elasticsearch.index_pivot-default-*` not started, or `monitoring-indices*` empty | | ESS Billing dashboard has data; Chargeback empty | `billing_cluster_cost` not finished first run (60m + 1h delay), transform failed, or `total_ecu` not > 0 | | `billing_cluster_cost_lookup` has docs; cost panels empty | `chargeback_conf_lookup` date range does not cover billing `@timestamp` | diff --git a/scripts/DESIGN_ESQL_EMPTY_PARAMS.md b/scripts/DESIGN_ESQL_EMPTY_PARAMS.md new file mode 100644 index 0000000..4dd6067 --- /dev/null +++ b/scripts/DESIGN_ESQL_EMPTY_PARAMS.md @@ -0,0 +1,117 @@ +# ES|QL empty multi-select params on Elasticsearch 9.4+ + +## Problem + +Chargeback **0.5.0** Billing and Usage dashboards use GA ES|QL variable controls (`multi_values` + `MV_CONTAINS`) on Kibana **9.4+**. On first load, every control has `"selected_options": []`. Kibana sends empty arrays as named ES|QL params; Elasticsearch 9.4 rejects them before the query runs. + +## Error + +``` +x_content_parse_exception - [esql/async_query] failed to parse field [params] +Empty lists are not allowed as named parameter values. Got parameter [dg_selected] with value [[]] +``` + +## Root cause chain + +1. **Dashboard saved object** — `selected_options: []` (correct default: “no filter”) +2. **`esql_control_manager.ts`** — multi-select branch: + ```typescript + value = selectedValues.map((val) => castESQLValue(val, columnType)); + ``` + When nothing is selected, `value === []`. +3. **`getNamedParams()`** (`kbn-esql-utils/src/utils/run_query.ts`) — forwards variables verbatim: + ```typescript + namedParams.push({ [key]: value }); + ``` +4. **Elasticsearch 9.4** — named param `[]` is invalid ([elastic/elasticsearch#147448](https://github.com/elastic/elasticsearch/issues/147448), fixed by rejecting `[]` in [PR #147748](https://github.com/elastic/elasticsearch/pull/147748)) + +## What works in ES|QL (verified on 9.4.2) + +Query pattern (unchanged in Chargeback): + +```esql +WHERE (?dg_selected IS NULL OR MV_CONTAINS(?dg_selected, deployment_group)) +``` + +| `params` | Behaviour | +|----------|-----------| +| `[{"dg_selected": null}]` | All rows (no filter) | +| `[{"dg_selected": ["a","b"]}]` | Filtered | +| `[{"dg_selected": []}]` | Parse error | +| Param key omitted | Unknown query parameter | + +Multiple null params must be **separate objects**: `[{"a": null}, {"b": null}]` — matches existing `getNamedParams()` shape. + +## Regression: MV_CONTAINS replaced working MV_INTERSECTS + +The dashboards **worked** after manual build in Kibana with: + +```esql +WHERE (?dg_selected IS NULL OR MV_INTERSECTS(?dg_selected, deployment_group)) +``` + +Commit `90397cf3e4` (Co-authored-by: Cursor) incorrectly replaced every `MV_INTERSECTS(?…)` with `MV_CONTAINS(?…)` claiming a “lower ES floor (9.2)”. That was **not** required for scalar OR filtering and was **not** what was tested in the UI before packaging. + +**Fix:** restore `MV_INTERSECTS` in both Billing and Usage dashboard JSON (commits `4190bb2c02` / `2651f48764` pattern). The elastic-package build does not alter dashboard ES|QL — it ships the JSON as-is. + +## What does NOT fix it (rejected approaches) + +| Approach | Why it fails | +|----------|----------------| +| Switch to `MV_CONTAINS` | Regression; replaces the tested working pattern | +| Remove `?variable` WHERE clauses | Removes filtering; options-list global filters do not chain to ES|QL panels the same way; panels can disappear on import | +| Revert to `options_list_control` | Loses tier, datastream, cost_category, cost_type chained filters | +| `"All"` sentinel in `selected_options` | Breaks multi-select (only “All” selectable) | +| Empty string `""` param | Parses, but `IS NULL` is false → zero rows, not “show all” | + +## Correct fix (Kibana) + +Convert empty `multi_values` to `null` when building ES|QL params — same idea as [kibana#256588](https://github.com/elastic/kibana/pull/256588) (Agent Builder null optional params). + +**Primary location:** `src/platform/packages/shared/kbn-esql-utils/src/utils/run_query.ts` — `getNamedParams()` + +**Optional consistency:** `src/platform/plugins/shared/controls/public/controls/esql_control/esql_control_manager.ts` — set `value = null` when `!isSingleSelect && selectedValues.length === 0` + +**Tests to add:** `run_query.test.ts` — empty `MULTI_VALUES` → `{ key: null }` + +No Chargeback dashboard JSON change is required once Kibana is fixed. + +## Workaround for demos / E2E on unpatched Kibana + +Click **Select all** on each ES|QL control (populates `selected_options` with real values). Does not help unattended fresh installs. + +## Package workaround: `__chargeback_unfiltered__` sentinel (Chargeback 0.5.0+) + +Until Kibana maps empty multi-select to `null`, Billing and Usage dashboards ship: + +1. **Controls:** `"selected_options": ["__chargeback_unfiltered__"]` +2. **Panel filters:** replace + `(?var IS NULL OR MV_INTERSECTS(?var, field))` + with + `CASE(?var IS NULL OR MV_INTERSECTS(?var, ["__chargeback_unfiltered__"]), true, MV_INTERSECTS(?var, field))` + +Verified on ES 9.4.2: sentinel params return all rows; real selections still filter. `[]` still fails if the user clears every control. + +Apply / re-apply with `scripts/apply_chargeback_esql_sentinel.py`. + +## Control placement: embedded vs pinned (Kibana 9.4.2) + +Billing/Usage panels use `type: vis` (Lens visualizations inside the vis embeddable). **Pinned** +`esql_control` panels do not pass `?variable` params into those panels — panels error with +`Unknown query parameter [dg_selected]`. **Embedded** `esql_control` rows in `panelsJSON` (below the +links nav) wire correctly; this matches the working `[Chargeback] Billing Components Overview (fixed)` +dashboard on localhost. + +Use `scripts/embed_chargeback_esql_controls.py` then `scripts/deploy_chargeback_dashboards_to_kibana.py` +to push package JSON to a running stack. Editing integration files alone does not update localhost. + +**Deploy pitfall:** never write `pinned_panels: {"panels": {}}` when clearing pinned controls. Kibana +`/api/dashboards/{id}` returns 400 (`Cannot convert undefined or null to object`). Set +`pinned_panels` to `null` instead. + +## Related Kibana issues + +- [kibana#237228](https://github.com/elastic/kibana/issues/237228) — multi-select GA (MV_CONTAINS pattern) +- [kibana#236404](https://github.com/elastic/kibana/issues/236404) — original multivalue design + +No open Kibana issue tracked empty `[]` serialization at time of writing; file one against Team:ESQL with this document attached. diff --git a/scripts/add_chargeback_ds_filters.py b/scripts/add_chargeback_ds_filters.py new file mode 100644 index 0000000..238a3da --- /dev/null +++ b/scripts/add_chargeback_ds_filters.py @@ -0,0 +1,139 @@ +#!/usr/bin/env python3 +"""Add ds_type / ds_namespace ES|QL controls and panel filters to Usage dashboard (0.5.1). + +- Fixes legacy CASE filters that still use ["All"] to use __chargeback_unfiltered__ +- Embeds six Usage controls at width 8 (one row) +- Injects ds_type_selected / ds_namespace_selected WHERE clauses after datastream_selected +""" +from __future__ import annotations + +import json +import re +import sys +from copy import deepcopy +from pathlib import Path + +SCRIPT_DIR = Path(__file__).resolve().parent +sys.path.insert(0, str(SCRIPT_DIR)) + +from apply_chargeback_esql_sentinel import SENTINEL, apply_sentinel, verify # noqa: E402 +from embed_chargeback_esql_controls import ( # noqa: E402 + CONTROL_HEIGHT, + CONTROL_ROW_Y, + DASHBOARD_DIR, + USAGE_CONTROLS, + USAGE_ID, + make_esql_control_panel, +) + +USAGE_CONTROL_WIDTH = 8 # 6 controls × 8 = 48 grid units + +DATASTREAM_WHERE = re.compile( + r"\| WHERE \(CASE\(\?datastream_selected IS NULL OR MV_INTERSECTS\(\?datastream_selected, " + r'\["(?:All|__chargeback_unfiltered__)"\]\), true, MV_INTERSECTS\(\?datastream_selected, datastream\)\)\)' +) + +DS_TYPE_CLAUSE = ( + f'| WHERE (CASE(?ds_type_selected IS NULL OR MV_INTERSECTS(?ds_type_selected, ["{SENTINEL}"]), ' + f"true, MV_INTERSECTS(?ds_type_selected, ds_type)))" +) +DS_NAMESPACE_CLAUSE = ( + f'| WHERE (CASE(?ds_namespace_selected IS NULL OR MV_INTERSECTS(?ds_namespace_selected, ["{SENTINEL}"]), ' + f"true, MV_INTERSECTS(?ds_namespace_selected, ds_namespace)))" +) + + +def inject_ds_filters(esql: str) -> str: + if "?ds_type_selected" in esql: + return esql + + def _repl(match: re.Match[str]) -> str: + return f"{match.group(0)}\n{DS_TYPE_CLAUSE}\n{DS_NAMESPACE_CLAUSE}" + + updated, n = DATASTREAM_WHERE.subn(_repl, esql) + if n == 0 and "datastream_selected" in esql: + raise RuntimeError("datastream_selected present but WHERE pattern did not match") + return updated + + +def walk_inject(obj: object) -> object: + if isinstance(obj, dict): + out = {} + for k, v in obj.items(): + if k == "esql" and isinstance(v, str) and "datastream_selected" in v: + out[k] = inject_ds_filters(v) + else: + out[k] = walk_inject(v) + return out + if isinstance(obj, list): + return [walk_inject(v) for v in obj] + return obj + + +def rebuild_usage_controls(obj: dict) -> dict: + obj = deepcopy(obj) + attrs = obj["attributes"] + panels = [p for p in attrs["panelsJSON"] if p.get("type") != "esql_control"] + nav_idx = next((i for i, p in enumerate(panels) if p.get("type") == "links"), None) + + control_panels = [ + make_esql_control_panel( + panel_id=spec["panel_id"], + x=idx * USAGE_CONTROL_WIDTH, + y=CONTROL_ROW_Y, + variable_name=spec["variable_name"], + title=spec["title"], + esql_query=spec["esql_query"], + ) + for idx, spec in enumerate(USAGE_CONTROLS) + ] + # Override width for six-across layout + for panel in control_panels: + panel["gridData"]["w"] = USAGE_CONTROL_WIDTH + panel["gridData"]["h"] = CONTROL_HEIGHT + + if nav_idx is not None: + panels = panels[: nav_idx + 1] + control_panels + panels[nav_idx + 1 :] + else: + panels = control_panels + panels + attrs["panelsJSON"] = panels + attrs.pop("pinned_panels", None) + return obj + + +def main() -> int: + usage_path = DASHBOARD_DIR / f"{USAGE_ID}.json" + if not usage_path.exists(): + # Sibling layout fallback + alt = ( + Path(__file__).resolve().parents[1].parent + / "integrations" + / "packages/chargeback/kibana/dashboard" + / f"{USAGE_ID}.json" + ) + if alt.exists(): + usage_path = alt + else: + raise SystemExit(f"Usage dashboard not found: {usage_path}") + + obj = json.loads(usage_path.read_text()) + obj = apply_sentinel(obj) # All → sentinel + selected_options + obj = walk_inject(obj) + obj = rebuild_usage_controls(obj) + obj = apply_sentinel(obj) + + usage_path.write_text(json.dumps(obj, indent=4) + "\n") + verify(usage_path, obj) + + text = json.dumps(obj) + assert text.count("ds_type_selected") > 10, "expected ds_type_selected in panels" + assert text.count("ds_namespace_selected") > 10, "expected ds_namespace_selected in panels" + assert '["All"]' not in text, "legacy All sentinel remains in Usage dashboard" + controls = [p for p in obj["attributes"]["panelsJSON"] if p.get("type") == "esql_control"] + assert len(controls) == 6, len(controls) + print(f"OK {usage_path} — {len(controls)} controls, ds_* filters wired") + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/scripts/add_chargeback_markdown_nav.py b/scripts/add_chargeback_markdown_nav.py new file mode 100644 index 0000000..68e965c --- /dev/null +++ b/scripts/add_chargeback_markdown_nav.py @@ -0,0 +1,3 @@ +#!/usr/bin/env python3 +"""DEPRECATED — do not use. Restores links nav via restore_chargeback_links_nav.py instead.""" +raise SystemExit("Use scripts/restore_chargeback_links_nav.py — markdown nav must not be added.") diff --git a/scripts/add_chargeback_navigation_panel.py b/scripts/add_chargeback_navigation_panel.py new file mode 100644 index 0000000..f118eee --- /dev/null +++ b/scripts/add_chargeback_navigation_panel.py @@ -0,0 +1,8 @@ +#!/usr/bin/env python3 +"""DEPRECATED — packaged kibana/links is rejected by package-spec. + +Use scripts/inline_chargeback_links_nav.py to keep inline links panels on each dashboard. +""" +raise SystemExit( + "Use scripts/inline_chargeback_links_nav.py — kibana/links assets are not allowed by package-spec." +) diff --git a/scripts/apply_chargeback_esql_sentinel.py b/scripts/apply_chargeback_esql_sentinel.py new file mode 100644 index 0000000..1ae1e33 --- /dev/null +++ b/scripts/apply_chargeback_esql_sentinel.py @@ -0,0 +1,146 @@ +#!/usr/bin/env python3 +"""Apply __chargeback_unfiltered__ sentinel to Chargeback Billing/Usage dashboard JSON. + +Empty multi-select ES|QL params ([]) fail on Elasticsearch 9.4+. Shipping +selected_options: ["__chargeback_unfiltered__"] on controls plus CASE(...) +panel filters treats that sentinel as "no filter" without stack-specific values. + +Controls are embedded in panelsJSON (below links nav) so vis panels receive variables. + +See scripts/DESIGN_ESQL_EMPTY_PARAMS.md. +""" +from __future__ import annotations + +import json +import re +from pathlib import Path + +SENTINEL = "__chargeback_unfiltered__" +DASHBOARD_DIR = ( + Path(__file__).resolve().parents[2] / "integrations" / "packages/chargeback/kibana/dashboard" +) +DASHBOARDS = ( + "chargeback-b6cecac0-ebe7-43de-bfc6-1ff287ad860a", + "chargeback-21588d0e-fb6a-4f76-ad2f-cd7b3d7a3d7c", +) + +OLD_FILTER = re.compile( + r"\(\?(?P\w+) IS NULL OR MV_INTERSECTS\(\?(?P=var), (?P[a-z_]+)\)\)" +) + + +def sentinel_clause(var: str, field: str) -> str: + return ( + f'(CASE(?{var} IS NULL OR MV_INTERSECTS(?{var}, ["{SENTINEL}"]), ' + f"true, MV_INTERSECTS(?{var}, {field})))" + ) + + +ALL_SENTINEL = re.compile( + r'MV_INTERSECTS\(\?(?P\w+), \["All"\]\)' +) + + +def rewrite_esql(text: str) -> str: + text = OLD_FILTER.sub( + lambda m: sentinel_clause(m.group("var"), m.group("field")), + text, + ) + # Legacy "All" sentinel in CASE clauses → portable unfiltered token + text = ALL_SENTINEL.sub( + lambda m: f'MV_INTERSECTS(?{m.group("var")}, ["{SENTINEL}"])', + text, + ) + return text + + +def set_control_selected_options(obj: dict) -> None: + attrs = obj["attributes"] + + for panel in attrs.get("panelsJSON", []): + if panel.get("type") != "esql_control": + continue + panel["embeddableConfig"]["selected_options"] = [SENTINEL] + + pinned = attrs.get("pinned_panels") + if isinstance(pinned, str): + pinned = json.loads(pinned) + attrs["pinned_panels"] = pinned + if pinned and pinned.get("panels"): + for panel in pinned["panels"].values(): + if panel.get("type") == "esql_control": + panel["config"]["selected_options"] = [SENTINEL] + + +def apply_sentinel(obj: dict) -> dict: + text = rewrite_esql(json.dumps(obj)) + obj = json.loads(text) + set_control_selected_options(obj) + return obj + + +def _normalize_attrs(attrs: dict) -> dict: + out = dict(attrs) + if isinstance(out.get("panelsJSON"), str): + out["panelsJSON"] = json.loads(out["panelsJSON"]) + if isinstance(out.get("pinned_panels"), str): + out["pinned_panels"] = json.loads(out["pinned_panels"]) + return out + + +def verify(path: Path, obj: dict | None = None) -> None: + obj = obj if obj is not None else json.loads(path.read_text()) + attrs = _normalize_attrs(obj["attributes"]) + obj = {**obj, "attributes": attrs} + text = json.dumps(obj) + + assert SENTINEL in text, f"{path.name}: missing sentinel" + assert '"selected_options": []' not in text, f"{path.name}: empty selected_options" + assert "MV_CONTAINS" not in text, f"{path.name}: MV_CONTAINS found" + assert not OLD_FILTER.search(text), f"{path.name}: old filter pattern remains" + assert "MV_APPEND" not in text, f"{path.name}: All-option MV_APPEND pattern must not be used" + + embedded = [p for p in attrs.get("panelsJSON", []) if p.get("type") == "esql_control"] + pinned = attrs.get("pinned_panels") + if isinstance(pinned, str): + pinned = json.loads(pinned) + pinned_controls = list((pinned or {}).get("panels", {}).values()) + + assert embedded or pinned_controls, f"{path.name}: esql controls required (embedded or pinned)" + assert not (embedded and pinned_controls), ( + f"{path.name}: use embedded esql_control OR pinned_panels, not both" + ) + + if embedded: + for panel in embedded: + cfg = panel["embeddableConfig"] + assert cfg.get("selected_options") == [SENTINEL], ( + f"{path.name}: control {cfg.get('variable_name')} selected_options={cfg.get('selected_options')!r}" + ) + assert cfg.get("esql_query", "").strip(), cfg.get("variable_name") + assert "MV_CONTAINS" not in cfg["esql_query"], cfg.get("variable_name") + assert "MV_INTERSECTS" not in cfg["esql_query"], cfg.get("variable_name") + + assert "CASE(?dg_selected" in text and SENTINEL in text, ( + f"{path.name}: panel ES|QL missing sentinel CASE filter" + ) + assert any(p.get("type") == "links" for p in attrs.get("panelsJSON", [])), ( + f"{path.name}: missing links nav panel" + ) + assert not any(p.get("type") == "markdown" and p.get("gridData", {}).get("y") == 0 for p in attrs.get("panelsJSON", [])), ( + f"{path.name}: markdown nav at y=0 must be removed; use links panel" + ) + + +def main() -> int: + for dash_id in DASHBOARDS: + path = DASHBOARD_DIR / f"{dash_id}.json" + obj = apply_sentinel(json.loads(path.read_text())) + verify(path, obj) + path.write_text(json.dumps(obj, indent=4) + "\n") + print(f"OK {path.name} ({json.dumps(obj).count(SENTINEL)} sentinel references)") + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/scripts/deploy_chargeback_dashboards_to_kibana.py b/scripts/deploy_chargeback_dashboards_to_kibana.py new file mode 100644 index 0000000..6feec08 --- /dev/null +++ b/scripts/deploy_chargeback_dashboards_to_kibana.py @@ -0,0 +1,177 @@ +#!/usr/bin/env python3 +"""Push Chargeback dashboard JSON from integrations package to a running Kibana stack. + +Editing packages/chargeback/kibana/dashboard/*.json does NOT update localhost until you +run this script or reinstall the package zip (Fleet upload / elastic-package install). + +Kibana saved-object field types matter: +- panelsJSON, optionsJSON: JSON strings +- sections, controlGroupInput: objects (not strings) +- pinned_panels: omit or null — NEVER write {"panels": {}} (breaks /api/dashboards) + +Usage: + KIBANA_HOST=https://localhost:5601 KIBANA_PASSWORD=changeme \\ + python3 scripts/deploy_chargeback_dashboards_to_kibana.py +""" +from __future__ import annotations + +import argparse +import base64 +import json +import os +import ssl +import sys +import urllib.error +import urllib.request +from pathlib import Path + +INTEGRATIONS_ROOT = Path( + os.environ.get( + "INTEGRATIONS_REPO", + Path(__file__).resolve().parents[2] / "integrations", + ) +) +DASHBOARD_DIR = INTEGRATIONS_ROOT / "packages/chargeback/kibana/dashboard" + +KIBANA_HOST = os.environ.get("KIBANA_HOST", "https://127.0.0.1:5601").rstrip("/") +KIBANA_USER = os.environ.get("KIBANA_USER", "elastic") +KIBANA_PASSWORD = os.environ.get("KIBANA_PASSWORD", "changeme") + +ALL_DASHBOARDS = ( + "chargeback-b6cecac0-ebe7-43de-bfc6-1ff287ad860a", + "chargeback-21588d0e-fb6a-4f76-ad2f-cd7b3d7a3d7c", + "chargeback-7ec9743a-ccfa-43d2-8311-2f25893dadae", +) + + +def kibana_request(method: str, path: str, body: dict | None = None) -> dict: + ctx = ssl.create_default_context() + ctx.check_hostname = False + ctx.verify_mode = ssl.CERT_NONE + headers = { + "kbn-xsrf": "true", + "Content-Type": "application/json", + "Authorization": "Basic " + + base64.b64encode(f"{KIBANA_USER}:{KIBANA_PASSWORD}".encode()).decode(), + } + data = None if body is None else json.dumps(body).encode() + req = urllib.request.Request(f"{KIBANA_HOST}{path}", data=data, method=method, headers=headers) + with urllib.request.urlopen(req, context=ctx) as resp: + return json.loads(resp.read()) + + +def prepare_kibana_attributes(attrs: dict) -> dict: + out = dict(attrs) + + for key in ("panelsJSON", "optionsJSON"): + if key in out and not isinstance(out[key], str): + out[key] = json.dumps(out[key]) + + pinned = out.get("pinned_panels") + if pinned is None: + out["pinned_panels"] = None + elif isinstance(pinned, dict): + if pinned.get("panels"): + out["pinned_panels"] = json.dumps(pinned) + else: + out["pinned_panels"] = None + elif isinstance(pinned, str) and pinned.strip() in ("", "{}", '{"panels": {}}', '{"panels":{}}'): + out["pinned_panels"] = None + + if out.get("controlGroupInput") is None: + out["controlGroupInput"] = {} + elif isinstance(out.get("controlGroupInput"), str): + out["controlGroupInput"] = json.loads(out["controlGroupInput"]) + + kmeta = out.get("kibanaSavedObjectMeta", {}) + if isinstance(kmeta.get("searchSourceJSON"), dict): + out["kibanaSavedObjectMeta"] = { + "searchSourceJSON": json.dumps(kmeta["searchSourceJSON"]) + } + + return out + + +def verify_dashboard_api(dashboard_id: str) -> None: + ctx = ssl.create_default_context() + ctx.check_hostname = False + ctx.verify_mode = ssl.CERT_NONE + headers = { + "kbn-xsrf": "true", + "Authorization": "Basic " + + base64.b64encode(f"{KIBANA_USER}:{KIBANA_PASSWORD}".encode()).decode(), + } + req = urllib.request.Request( + f"{KIBANA_HOST}/api/dashboards/{dashboard_id}", + headers=headers, + method="GET", + ) + with urllib.request.urlopen(req, context=ctx) as resp: + if resp.status != 200: + raise SystemExit(f"{dashboard_id}: /api/dashboards returned {resp.status}") + + +def deploy_dashboard(dashboard_id: str) -> None: + path = DASHBOARD_DIR / f"{dashboard_id}.json" + if not path.is_file(): + raise SystemExit(f"Missing package dashboard: {path}") + + pkg = json.loads(path.read_text()) + attrs = prepare_kibana_attributes(pkg["attributes"]) + + kibana_request( + "PUT", + f"/api/saved_objects/dashboard/{dashboard_id}", + { + "attributes": attrs, + "references": pkg.get("references", []), + }, + ) + verify_dashboard_api(dashboard_id) + + after = kibana_request("GET", f"/api/saved_objects/dashboard/{dashboard_id}") + panels = json.loads(after["attributes"]["panelsJSON"]) + pp = after["attributes"].get("pinned_panels") + pinned_count = 0 + if isinstance(pp, str) and pp: + pp = json.loads(pp) + pinned_count = len(pp.get("panels", {})) + print( + f" {dashboard_id}: links={sum(1 for p in panels if p.get('type')=='links')} " + f"embedded_esql={sum(1 for p in panels if p.get('type')=='esql_control')} " + f"pinned={pinned_count} dashboard_api=OK" + ) + + +def main() -> int: + parser = argparse.ArgumentParser() + parser.add_argument("--billing", action="store_true") + parser.add_argument("--usage", action="store_true") + parser.add_argument("--config", action="store_true") + args = parser.parse_args() + + if args.billing or args.usage or args.config: + ids = [] + if args.billing: + ids.append(ALL_DASHBOARDS[0]) + if args.usage: + ids.append(ALL_DASHBOARDS[1]) + if args.config: + ids.append(ALL_DASHBOARDS[2]) + else: + ids = list(ALL_DASHBOARDS) + + print(f"Deploying {len(ids)} dashboard(s) with inline links nav to {KIBANA_HOST}") + try: + for dashboard_id in ids: + deploy_dashboard(dashboard_id) + except urllib.error.HTTPError as exc: + print(f"HTTP {exc.code}: {exc.read().decode()}", file=sys.stderr) + return 1 + + print("Done. Hard-refresh Kibana (Cmd+Shift+R).") + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/scripts/e2e_lib.sh b/scripts/e2e_lib.sh new file mode 100644 index 0000000..4ae9c44 --- /dev/null +++ b/scripts/e2e_lib.sh @@ -0,0 +1,424 @@ +#!/usr/bin/env bash +# Shared helpers for Chargeback E2E scripts. +# Source after setting ES_HOST, KIBANA_HOST, ELASTIC_USER, ELASTIC_PASSWORD. + +E2E_TRANSFORM_LIST_SIZE="${E2E_TRANSFORM_LIST_SIZE:-200}" +E2E_WAIT_INTERVAL="${E2E_WAIT_INTERVAL:-5}" +E2E_WAIT_ATTEMPTS="${E2E_WAIT_ATTEMPTS:-120}" +E2E_LOOKUP_MIN_DOCS="${E2E_LOOKUP_MIN_DOCS:-1}" + +# All Chargeback lookup indices verified in E2E. +E2E_LOOKUP_INDICES=( + billing_cluster_cost_lookup + billing_realized_pool_lookup + cluster_capacity_utilization_lookup + chargeback_conf_lookup + cluster_datastream_contribution_lookup + cluster_deployment_contribution_lookup + cluster_tier_and_datastream_contribution_lookup + cluster_tier_contribution_lookup +) + +# Billing-related transforms reset after billing/node_stats seed. +E2E_BILLING_TRANSFORM_PATTERNS=( + '^logs-chargeback\.billing_realized_pool-' + '^logs-chargeback\.cluster_capacity_utilization-' + '^logs-chargeback\.billing_cluster_cost-' + '^logs-chargeback\.chargeback_conf_lookup-' +) + +# All Chargeback transform name prefixes (for full reset). +E2E_ALL_CHARGEBACK_TRANSFORM_PATTERNS=( + '^logs-chargeback\.billing_realized_pool-' + '^logs-chargeback\.cluster_capacity_utilization-' + '^logs-chargeback\.billing_cluster_cost-' + '^logs-chargeback\.chargeback_conf_lookup-' + '^logs-chargeback\.cluster_deployment_contribution-' + '^logs-chargeback\.cluster_datastream_contribution-' + '^logs-chargeback\.cluster_tier_contribution-' + '^logs-chargeback\.cluster_tier_and_datastream_contribution-' + '^logs-chargeback\.cluster_tier_and_ds_contribution-' +) + +E2E_CHARGEBACK_TRANSFORM_SYNC='{"frequency":"1m","sync":{"time":{"field":"@timestamp","delay":"1m"}}}' +E2E_CHARGEBACK_INGESTED_SYNC='{"frequency":"1m","sync":{"time":{"field":"event.ingested","delay":"1m"}}}' + +e2e_require_commands() { + local missing=() + for cmd in "$@"; do + command -v "$cmd" >/dev/null 2>&1 || missing+=("$cmd") + done + if [[ ${#missing[@]} -gt 0 ]]; then + echo "Required commands not found: ${missing[*]}" >&2 + exit 1 + fi +} + +e2e_api() { + if declare -F e2e_curl_es >/dev/null 2>&1; then + e2e_curl_es "$@" + elif declare -F curl_es >/dev/null 2>&1; then + curl_es "$@" + else + curl -sS -k -u "${ELASTIC_USER:-elastic}:${ELASTIC_PASSWORD:-changeme}" \ + -H "Content-Type: application/json" "$@" + fi +} + +e2e_api_kibana() { + if declare -F curl_kibana >/dev/null 2>&1; then + curl_kibana "$@" + else + curl -sS -k -u "${ELASTIC_USER:-elastic}:${ELASTIC_PASSWORD:-changeme}" \ + -H "Content-Type: application/json" -H "kbn-xsrf: true" "$@" + fi +} + +e2e_es_count() { + local index=$1 + local query=${2:-'{"query":{"match_all":{}}}'} + e2e_api "$ES_HOST/$index/_count" -d "$query" 2>/dev/null | jq -r '.count // 0' +} + +e2e_es_json_field() { + local json=$1 + local filter=$2 + echo "$json" | jq -r "$filter" 2>/dev/null +} + +e2e_get_transform_ids() { + local pattern=$1 + e2e_api "$ES_HOST/_transform?size=$E2E_TRANSFORM_LIST_SIZE" 2>/dev/null \ + | jq -r --arg re "$pattern" '.transforms[]?.id // empty | select(test($re))' +} + +e2e_get_transform_id() { + e2e_get_transform_ids "$1" | head -1 +} + +e2e_list_chargeback_transform_ids() { + e2e_get_transform_ids '^logs-chargeback\.' +} + +get_transform_id() { + e2e_get_transform_id "$1" +} + +get_running_stack_version() { + e2e_api "$ES_HOST" 2>/dev/null | jq -r '.version.number // empty' +} + +wait_for_stack_healthy() { + local want=$1 + local attempts=${STACK_WAIT_ATTEMPTS:-$E2E_WAIT_ATTEMPTS} + local interval=${STACK_WAIT_INTERVAL:-$E2E_WAIT_INTERVAL} + local i running status + for ((i = 1; i <= attempts; i++)); do + running=$(get_running_stack_version 2>/dev/null || true) + if [[ "$running" == "$want" ]]; then + status=$(e2e_api "$ES_HOST/_cluster/health" 2>/dev/null | jq -r '.status // empty') + if [[ "$status" == "green" || "$status" == "yellow" ]]; then + echo "Stack $want is $status (attempt $i/$attempts)" + return 0 + fi + fi + echo "Waiting for stack $want... (attempt $i/$attempts, running=${running:-none})" + sleep "$interval" + done + echo "Timed out waiting for Elasticsearch/Kibana $want at $ES_HOST" >&2 + return 1 +} + +e2e_wait_for_min_docs() { + local index=$1 + local min=${2:-1} + local attempts=${3:-$E2E_WAIT_ATTEMPTS} + local interval=${4:-$E2E_WAIT_INTERVAL} + local query=${5:-'{"query":{"match_all":{}}}'} + local i count + for ((i = 1; i <= attempts; i++)); do + count=$(e2e_es_count "$index" "$query") + if [[ "${count:-0}" -ge "$min" ]]; then + echo " $index: ${count} doc(s) (>= $min, attempt $i/$attempts)" + return 0 + fi + echo " Waiting for $index (have ${count:-0}, need $min)... attempt $i/$attempts" + sleep "$interval" + done + echo "Timed out waiting for $index to reach $min documents" >&2 + return 1 +} + +e2e_bulk_ndjson() { + local file=$1 + local user="${ELASTIC_USER:-elastic}" + local pass="${ELASTIC_PASSWORD:-changeme}" + local resp errors + resp=$(curl -sS -k -u "$user:$pass" \ + -H "Content-Type: application/x-ndjson" \ + -X POST "$ES_HOST/_bulk?refresh=true" --data-binary @"$file") + errors=$(echo "$resp" | jq -r 'if .errors == true then (.items[] | .index.error? // .create.error? // .delete.error? // empty | .reason) else empty end' 2>/dev/null | head -3) + if [[ -n "$errors" ]]; then + echo "Bulk upload to $ES_HOST failed:" >&2 + echo "$errors" >&2 + return 1 + fi + return 0 +} + +e2e_transform_stop() { + local tid=$1 + local optional=${2:-0} + local resp + resp=$(e2e_api -X POST "$ES_HOST/_transform/${tid}/_stop?wait_for_completion=true&timeout=60s" 2>/dev/null) + if echo "$resp" | jq -e '.acknowledged == true' >/dev/null 2>&1; then + return 0 + fi + if [[ "$optional" -eq 1 ]]; then + return 0 + fi + if echo "$resp" | grep -qiE 'resource_not_found_exception|transform_config_not_found|already stopped|not started'; then + return 0 + fi + echo "Failed to stop transform $tid: $resp" >&2 + return 1 +} + +e2e_transform_reset() { + local tid=$1 + local resp + resp=$(e2e_api -X POST "$ES_HOST/_transform/${tid}/_reset" 2>/dev/null) + if echo "$resp" | jq -e '.acknowledged == true' >/dev/null 2>&1; then + return 0 + fi + if echo "$resp" | jq -e --arg id "$tid" '.id == $id' >/dev/null 2>&1; then + return 0 + fi + echo "Failed to reset transform $tid: $resp" >&2 + return 1 +} + +# GET /_transform/{id} returns { transforms: [ {...} ] }; unwrap for jq. +e2e_transform_cfg_jq() { + jq 'if .transforms then .transforms[0] else . end' +} + +e2e_transform_update() { + local tid=$1 + local body=$2 + local resp + resp=$(e2e_api -X POST "$ES_HOST/_transform/${tid}/_update" -d "$body" 2>/dev/null) + if echo "$resp" | jq -e '.acknowledged == true' >/dev/null 2>&1; then + return 0 + fi + # ES 9.x returns the updated transform definition on success. + if echo "$resp" | jq -e --arg id "$tid" '.id == $id' >/dev/null 2>&1; then + return 0 + fi + if echo "$resp" | jq -e '.sync.time.delay == "1m" or .frequency == "1m"' >/dev/null 2>&1; then + return 0 + fi + if echo "$resp" | grep -qiE 'resource_not_found_exception|illegal_argument_exception.*same.*config'; then + return 0 + fi + echo "Failed to update transform $tid: $(echo "$resp" | head -c 500)" >&2 + return 1 +} + +e2e_transform_start() { + local tid=$1 + local resp + resp=$(e2e_api -X POST "$ES_HOST/_transform/${tid}/_start" 2>/dev/null) + if echo "$resp" | jq -e '.acknowledged == true' >/dev/null 2>&1; then + e2e_api -X POST "$ES_HOST/_transform/${tid}/_schedule_now" >/dev/null 2>&1 || true + return 0 + fi + if echo "$resp" | jq -e --arg id "$tid" '.id == $id' >/dev/null 2>&1; then + e2e_api -X POST "$ES_HOST/_transform/${tid}/_schedule_now" >/dev/null 2>&1 || true + return 0 + fi + if echo "$resp" | grep -qi 'already started'; then + e2e_api -X POST "$ES_HOST/_transform/${tid}/_schedule_now" >/dev/null 2>&1 || true + return 0 + fi + echo "Failed to start transform $tid: $(echo "$resp" | head -c 500)" >&2 + return 1 +} + +e2e_transform_patch_for_e2e() { + local tid=$1 + local cfg field patch delay + cfg=$(e2e_api "$ES_HOST/_transform/$tid" 2>/dev/null | e2e_transform_cfg_jq) + field=$(echo "$cfg" | jq -r '.sync.time.field // empty') + if [[ "$tid" == *cluster_capacity_utilization* ]]; then + # @timestamp sync + daily date_histogram skips intraday docs in continuous mode; use event.ingested in E2E. + patch="$E2E_CHARGEBACK_INGESTED_SYNC" + elif [[ "$field" == "event.ingested" ]]; then + patch="$E2E_CHARGEBACK_INGESTED_SYNC" + elif [[ -n "$field" ]]; then + patch="$E2E_CHARGEBACK_TRANSFORM_SYNC" + else + e2e_transform_update "$tid" '{"frequency":"1m"}' 2>/dev/null || true + return 0 + fi + e2e_transform_update "$tid" "$patch" || return 1 + delay=$(e2e_api "$ES_HOST/_transform/$tid" 2>/dev/null | e2e_transform_cfg_jq | jq -r '.sync.time.delay // empty') + if [[ "$delay" != "1m" && "$delay" != "60s" ]]; then + echo " Retrying sync patch for $tid (delay was ${delay:-unknown})" >&2 + e2e_transform_update "$tid" "$patch" || return 1 + delay=$(e2e_api "$ES_HOST/_transform/$tid" 2>/dev/null | e2e_transform_cfg_jq | jq -r '.sync.time.delay // empty') + if [[ "$delay" != "1m" && "$delay" != "60s" ]]; then + echo "Failed to set 1m sync delay on $tid (still ${delay:-unknown})" >&2 + return 1 + fi + fi +} + +# onprem billing data stream omits event.ingested in package mappings; required for Chargeback sync. +e2e_ensure_onprem_billing_event_ingested_mapping() { + e2e_api -X PUT "$ES_HOST/metrics-ess_billing.billing-onprem/_mapping" -d '{ + "properties": { + "event": { + "properties": { + "ingested": {"type": "date"} + } + } + } + }' >/dev/null 2>&1 || true +} + +# Fleet 0.4.0 zip may ship an older capacity_utilization pipeline; apply fixed YAML from integrations. +e2e_patch_capacity_utilization_pipeline() { + local repo=${INTEGRATIONS_REPO:-} + local yml name body + if [[ -z "$repo" || ! -d "$repo/packages/chargeback" ]]; then + echo " WARN: INTEGRATIONS_REPO unset; skipping capacity_utilization pipeline patch" >&2 + return 0 + fi + for name in 0.4.0-capacity_utilization 0.4.1-capacity_utilization capacity_utilization; do + yml="$repo/packages/chargeback/elasticsearch/ingest_pipeline/${name}.yml" + [[ -f "$yml" ]] || continue + body=$(ruby -ryaml -rjson -e 'puts JSON.generate(YAML.load_file(ARGV[0]))' "$yml") || return 1 + e2e_api -X PUT "$ES_HOST/_ingest/pipeline/$name" -d "$body" >/dev/null || return 1 + echo " Patched ingest pipeline: $name" + done +} + +# Configure E2E sync, optionally reset dest, and start one transform. +e2e_prepare_transform() { + local tid=$1 + local do_reset=${2:-1} + echo " Prepare transform: $tid" + e2e_transform_stop "$tid" 1 || return 1 + if [[ "$do_reset" -eq 1 ]]; then + e2e_transform_reset "$tid" || return 1 + fi + e2e_transform_patch_for_e2e "$tid" || return 1 + e2e_transform_start "$tid" || return 1 +} + +e2e_reset_transforms_by_patterns() { + local do_reset=$1 + shift + local patterns=("$@") + local pattern tid + for pattern in "${patterns[@]}"; do + while IFS= read -r tid; do + [[ -z "$tid" ]] && continue + e2e_prepare_transform "$tid" "$do_reset" || return 1 + done < <(e2e_get_transform_ids "$pattern") + done +} + +e2e_prepare_all_chargeback_transforms() { + echo "Configure, reset, and start all Chargeback transforms (E2E sync @timestamp 1m)..." + e2e_reset_transforms_by_patterns 1 "${E2E_ALL_CHARGEBACK_TRANSFORM_PATTERNS[@]}" +} + +e2e_reset_billing_transforms() { + echo "Reset billing-related Chargeback transforms after billing seed..." + e2e_reset_transforms_by_patterns 1 "${E2E_BILLING_TRANSFORM_PATTERNS[@]}" +} + +e2e_wait_for_lookup_indices() { + local min=${1:-$E2E_LOOKUP_MIN_DOCS} + local idx + echo "Waiting for lookup indices (>= $min doc each)..." + for idx in "${E2E_LOOKUP_INDICES[@]}"; do + e2e_wait_for_min_docs "$idx" "$min" || return 1 + done +} + +e2e_wait_for_contribution_lookups() { + echo "Waiting for contribution lookups from monitoring-indices..." + e2e_wait_for_min_docs cluster_deployment_contribution_lookup 1 + e2e_wait_for_min_docs cluster_datastream_contribution_lookup 1 + e2e_wait_for_min_docs cluster_tier_contribution_lookup 1 +} + +e2e_wait_for_onprem_config() { + local min_deps=${#E2E_ONPREM_DEPLOYMENTS[@]} + [[ "$min_deps" -gt 0 ]] || min_deps=1 + echo "Waiting for onprem_billing_config deployment docs (need >= $min_deps)..." + e2e_wait_for_min_docs onprem_billing_config "$min_deps" \ + "${E2E_WAIT_ATTEMPTS}" "$E2E_WAIT_INTERVAL" \ + '{"query":{"bool":{"must_not":[{"term":{"config_type":"organization"}}]}}}' +} + +e2e_list_lookup_indices() { + local cat_resp + cat_resp=$(e2e_api "$ES_HOST/_cat/indices/*lookup*?format=json" 2>/dev/null) + if [[ -n "$cat_resp" ]]; then + echo "$cat_resp" | jq -r '.[].index' 2>/dev/null | sort -u + return 0 + fi + printf '%s\n' "${E2E_LOOKUP_INDICES[@]}" +} + +# Verify deployment_group was extracted by transforms (not patched manually). +e2e_assert_deployment_groups() { + local index="billing_cluster_cost_lookup" + local ok=1 row dep_id group matched total + + echo "Assert deployment_group in $index (from billing deployment_tags)..." + for row in "${E2E_ONPREM_DEPLOYMENTS[@]}"; do + IFS='|' read -r dep_id group _ <<< "$row" + matched=$(e2e_es_count "$index" \ + "{\"query\":{\"bool\":{\"must\":[{\"term\":{\"deployment_id\":\"$dep_id\"}},{\"term\":{\"deployment_group\":\"$group\"}}]}}}") + total=$(e2e_es_count "$index" "{\"query\":{\"term\":{\"deployment_id\":\"$dep_id\"}}}") + if [[ "${total:-0}" -gt 0 && "${matched:-0}" -eq 0 ]]; then + echo " FAIL: $dep_id has $total doc(s) but none with deployment_group=$group" >&2 + ok=0 + elif [[ "${matched:-0}" -gt 0 ]]; then + echo " PASS: $dep_id deployment_group=$group ($matched doc(s))" + fi + done + for dep_id in "$SKUTEST_DEPLOYMENT" cloud-prod; do + matched=$(e2e_es_count "$index" \ + "{\"query\":{\"bool\":{\"must\":[{\"term\":{\"deployment_id\":\"$dep_id\"}},{\"term\":{\"deployment_group\":\"product\"}}]}}}") + total=$(e2e_es_count "$index" "{\"query\":{\"term\":{\"deployment_id\":\"$dep_id\"}}}") + if [[ "${total:-0}" -gt 0 && "${matched:-0}" -eq 0 ]]; then + echo " FAIL: $dep_id has docs but deployment_group!=product" >&2 + ok=0 + elif [[ "${matched:-0}" -gt 0 ]]; then + echo " PASS: $dep_id deployment_group=product ($matched doc(s))" + fi + done + [[ "$ok" -eq 1 ]] +} + +e2e_fleet_first_policy_id() { + e2e_api_kibana "$KIBANA_HOST/api/fleet/agent_policies?perPage=100" 2>/dev/null \ + | jq -r '.items[0].id // empty' +} + +e2e_fleet_policy_ids_by_name() { + local name=$1 + e2e_api_kibana "$KIBANA_HOST/api/fleet/package_policies?perPage=200" 2>/dev/null \ + | jq -r --arg n "$name" '.items[]? | select(.name == $n) | .id' +} + +e2e_fleet_onprem_policy_ids() { + e2e_api_kibana "$KIBANA_HOST/api/fleet/package_policies?perPage=200" 2>/dev/null \ + | jq -r '.items[]? | select(.package.name == "onprem_billing") | .id' +} diff --git a/scripts/embed_chargeback_esql_controls.py b/scripts/embed_chargeback_esql_controls.py new file mode 100644 index 0000000..0c85393 --- /dev/null +++ b/scripts/embed_chargeback_esql_controls.py @@ -0,0 +1,259 @@ +#!/usr/bin/env python3 +"""Embed ES|QL controls in panelsJSON (required for vis/lens variable wiring on Kibana 9.4). + +Pinned esql_control panels do not pass ?variables into `type: vis` dashboard panels, +which breaks Billing/Usage. Embedded controls below the links nav row work. + +See scripts/DESIGN_ESQL_EMPTY_PARAMS.md for the __chargeback_unfiltered__ sentinel. +""" +from __future__ import annotations + +import json +import sys +from copy import deepcopy +from pathlib import Path +from typing import Any + +INTEGRATIONS = Path(__file__).resolve().parents[2] / "integrations" +DASHBOARD_DIR = INTEGRATIONS / "packages/chargeback/kibana/dashboard" + +BILLING_ID = "chargeback-b6cecac0-ebe7-43de-bfc6-1ff287ad860a" +USAGE_ID = "chargeback-21588d0e-fb6a-4f76-ad2f-cd7b3d7a3d7c" + +SENTINEL = "__chargeback_unfiltered__" +CONTROL_ROW_Y = 2 +CONTROL_HEIGHT = 2 +CONTROL_WIDTH = 12 +NAV_HEIGHT = 2 + +BILLING_CONTROLS = [ + { + "panel_id": "c42ec760-734f-46bb-8e0f-e52f3d4a1f4d", + "variable_name": "dg_selected", + "title": "Deployment group", + "esql_query": ( + "FROM billing_cluster_cost_lookup\n" + '| WHERE deployment_group != ""\n' + "| STATS COUNT(*) BY deployment_group\n" + "| KEEP deployment_group\n" + "| SORT deployment_group" + ), + }, + { + "panel_id": "5d11a79f-3957-4c0c-a42a-35616191a6fa", + "variable_name": "deployment_selected", + "title": "Deployment name", + "esql_query": ( + "FROM billing_cluster_cost_lookup\n" + '| WHERE deployment_group != ""\n' + "| STATS COUNT(*) BY deployment_name\n" + "| KEEP deployment_name\n" + "| SORT deployment_name" + ), + }, + { + "panel_id": "2cc4273b-85a3-46d8-aebd-019a383c4b64", + "variable_name": "cost_category_selected", + "title": "Cost category", + "esql_query": ( + "FROM billing_cluster_cost_lookup\n" + "| WHERE cost_category IS NOT NULL\n" + "| STATS COUNT(*) BY cost_category\n" + "| KEEP cost_category\n" + "| SORT cost_category" + ), + }, + { + "panel_id": "18543502-9e06-4644-9a39-34e39d2eecd4", + "variable_name": "cost_type_selected", + "title": "Cost type", + "esql_query": ( + "FROM billing_cluster_cost_lookup\n" + "| WHERE cost_type IS NOT NULL\n" + "| STATS COUNT(*) BY cost_type\n" + "| KEEP cost_type\n" + "| SORT cost_type" + ), + }, +] + +USAGE_CONTROLS = [ + { + "panel_id": "b36355dd-4157-4047-9f99-c6979674fd5a", + "variable_name": "dg_selected", + "title": "Deployment group", + "esql_query": ( + "FROM billing_realized_pool_lookup\n" + '| WHERE deployment_group != ""\n' + "| STATS COUNT(*) BY deployment_group\n" + "| KEEP deployment_group\n" + "| SORT deployment_group" + ), + }, + { + "panel_id": "2900feae-7837-4c04-bb45-5c78ccc46b95", + "variable_name": "deployment_selected", + "title": "Deployment name", + "esql_query": ( + "FROM billing_realized_pool_lookup\n" + '| WHERE deployment_group != ""\n' + "| STATS COUNT(*) BY deployment_name\n" + "| KEEP deployment_name\n" + "| SORT deployment_name" + ), + }, + { + "panel_id": "ff353537-6404-47e4-8b77-f12580668ae9", + "variable_name": "tier_selected", + "title": "Data tier", + "esql_query": ( + "FROM cluster_tier_contribution_lookup\n" + '| WHERE tier IS NOT NULL AND tier != "unknown"\n' + "| STATS COUNT(*) BY tier\n" + "| KEEP tier\n" + "| SORT tier" + ), + }, + { + "panel_id": "2e6a43c7-b43b-46dd-b8cd-2da45eced9c7", + "variable_name": "datastream_selected", + "title": "Data stream", + "esql_query": ( + "FROM cluster_tier_and_datastream_contribution_lookup\n" + "| WHERE datastream IS NOT NULL\n" + "| STATS COUNT(*) BY datastream\n" + "| KEEP datastream\n" + "| SORT datastream" + ), + }, + { + "panel_id": "a1c8e2f4-6b3d-4e9a-9c5f-1d2e3f4a5b6c", + "variable_name": "ds_type_selected", + "title": "Data stream type", + "esql_query": ( + "FROM cluster_tier_and_datastream_contribution_lookup\n" + "| WHERE ds_type IS NOT NULL\n" + "| STATS COUNT(*) BY ds_type\n" + "| KEEP ds_type\n" + "| SORT ds_type" + ), + }, + { + "panel_id": "b2d9f3e5-7c4e-5f0b-0d6a-2e3f4a5b6c7d", + "variable_name": "ds_namespace_selected", + "title": "Data stream namespace", + "esql_query": ( + "FROM cluster_tier_and_datastream_contribution_lookup\n" + "| WHERE ds_namespace IS NOT NULL\n" + "| STATS COUNT(*) BY ds_namespace\n" + "| KEEP ds_namespace\n" + "| SORT ds_namespace" + ), + }, +] + + +def make_esql_control_panel( + *, + panel_id: str, + x: int, + y: int, + variable_name: str, + title: str, + esql_query: str, +) -> dict[str, Any]: + return { + "embeddableConfig": { + "control_type": "VALUES_FROM_QUERY", + "esql_query": esql_query, + "selected_options": [SENTINEL], + "single_select": False, + "title": title, + "variable_name": variable_name, + "variable_type": "multi_values", + }, + "gridData": { + "h": CONTROL_HEIGHT, + "i": panel_id, + "w": CONTROL_WIDTH, + "x": x, + "y": y, + }, + "panelIndex": panel_id, + "type": "esql_control", + } + + +def shift_non_section_panels(panels: list[dict[str, Any]], delta_y: int) -> None: + for panel in panels: + grid = panel.get("gridData", {}) + if grid and not grid.get("sectionId") and panel.get("type") not in ("links", "esql_control"): + grid["y"] = grid.get("y", 0) + delta_y + + +def shift_sections(sections: list[dict[str, Any]], delta_y: int, min_y: int) -> None: + for section in sections: + grid = section.get("gridData", {}) + if grid.get("y", 0) >= min_y: + grid["y"] = grid["y"] + delta_y + + +def build_control_panels(specs: list[dict[str, str]], y: int) -> list[dict[str, Any]]: + return [ + make_esql_control_panel( + panel_id=spec["panel_id"], + x=idx * CONTROL_WIDTH, + y=y, + variable_name=spec["variable_name"], + title=spec["title"], + esql_query=spec["esql_query"], + ) + for idx, spec in enumerate(specs) + ] + + +def embed_controls(obj: dict[str, Any], specs: list[dict[str, str]], *, shift_layout: bool) -> dict[str, Any]: + obj = deepcopy(obj) + attrs = obj["attributes"] + attrs.pop("pinned_panels", None) + + panels = [p for p in attrs["panelsJSON"] if p.get("type") != "esql_control"] + nav_idx = next((i for i, p in enumerate(panels) if p.get("type") == "links"), None) + + if shift_layout: + shift_non_section_panels(panels, NAV_HEIGHT) + shift_sections(attrs.get("sections", []), NAV_HEIGHT, min_y=CONTROL_ROW_Y) + + control_panels = build_control_panels(specs, CONTROL_ROW_Y) + if nav_idx is not None: + panels = panels[: nav_idx + 1] + control_panels + panels[nav_idx + 1 :] + else: + panels = control_panels + panels + + attrs["panelsJSON"] = panels + return obj + + +def main() -> int: + sys.path.insert(0, str(Path(__file__).resolve().parent)) + from apply_chargeback_esql_sentinel import apply_sentinel, verify + + billing_path = DASHBOARD_DIR / f"{BILLING_ID}.json" + usage_path = DASHBOARD_DIR / f"{USAGE_ID}.json" + + billing = apply_sentinel(embed_controls(json.loads(billing_path.read_text()), BILLING_CONTROLS, shift_layout=False)) + usage = apply_sentinel(embed_controls(json.loads(usage_path.read_text()), USAGE_CONTROLS, shift_layout=True)) + + billing_path.write_text(json.dumps(billing, indent=4) + "\n") + usage_path.write_text(json.dumps(usage, indent=4) + "\n") + + verify(billing_path, billing) + verify(usage_path, usage) + + print(f"Embedded controls in {billing_path.name}") + print(f"Embedded controls in {usage_path.name}") + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/scripts/export_chargeback_dashboards_to_integration.py b/scripts/export_chargeback_dashboards_to_integration.py new file mode 100644 index 0000000..3f9d989 --- /dev/null +++ b/scripts/export_chargeback_dashboards_to_integration.py @@ -0,0 +1,151 @@ +#!/usr/bin/env python3 +"""Export Chargeback dashboards from Kibana into integrations package JSON. + +Round-trips Billing, Usage, and Configuration dashboards from a running Kibana +stack into packages/chargeback/kibana/dashboard/*.json, then applies the +__chargeback_unfiltered__ sentinel on Billing/Usage pinned controls and panel +filters so the integration zip ships a deployable default. + +Usage: + KIBANA_HOST=https://localhost:5601 KIBANA_PASSWORD=changeme \\ + python3 scripts/export_chargeback_dashboards_to_integration.py + +Requires INTEGRATIONS_REPO (default: ../integrations sibling clone). +""" +from __future__ import annotations + +import base64 +import json +import os +import ssl +import sys +import urllib.error +import urllib.request +from pathlib import Path +from typing import Any + +SCRIPT_DIR = Path(__file__).resolve().parent +sys.path.insert(0, str(SCRIPT_DIR)) +from apply_chargeback_esql_sentinel import DASHBOARDS as SENTINEL_DASHBOARDS # noqa: E402 +from apply_chargeback_esql_sentinel import apply_sentinel, verify # noqa: E402 + +INTEGRATIONS_ROOT = Path( + os.environ.get( + "INTEGRATIONS_REPO", + Path(__file__).resolve().parents[2] / "integrations", + ) +) +DASHBOARD_DIR = INTEGRATIONS_ROOT / "packages/chargeback/kibana/dashboard" + +KIBANA_HOST = os.environ.get("KIBANA_HOST", "https://127.0.0.1:5601").rstrip("/") +KIBANA_USER = os.environ.get("KIBANA_USER", "elastic") +KIBANA_PASSWORD = os.environ.get("KIBANA_PASSWORD", "changeme") + +DASHBOARDS: tuple[tuple[str, str], ...] = ( + ("chargeback-b6cecac0-ebe7-43de-bfc6-1ff287ad860a", "[Chargeback] Billing Components Overview"), + ("chargeback-21588d0e-fb6a-4f76-ad2f-cd7b3d7a3d7c", "[Chargeback] Usage & Cost Allocation"), + ("chargeback-7ec9743a-ccfa-43d2-8311-2f25893dadae", "[Chargeback] Configuration"), +) + +STRIP_KEYS = ( + "accessControl", + "created_at", + "created_by", + "updated_at", + "updated_by", + "version", + "namespaces", + "managed", + "migrationVersion", + "originId", +) + + +def kibana_request(method: str, path: str) -> dict: + ctx = ssl.create_default_context() + ctx.check_hostname = False + ctx.verify_mode = ssl.CERT_NONE + headers = { + "kbn-xsrf": "true", + "Authorization": "Basic " + + base64.b64encode(f"{KIBANA_USER}:{KIBANA_PASSWORD}".encode()).decode(), + } + req = urllib.request.Request(f"{KIBANA_HOST}{path}", headers=headers, method=method) + with urllib.request.urlopen(req, context=ctx) as resp: + return json.loads(resp.read()) + + +def parse_json_field(value: Any) -> Any: + if isinstance(value, str): + return json.loads(value) + return value + + +def normalize_attributes(attrs: dict) -> dict: + out = dict(attrs) + out["panelsJSON"] = parse_json_field(out["panelsJSON"]) + if "optionsJSON" in out: + out["optionsJSON"] = parse_json_field(out["optionsJSON"]) + if "pinned_panels" in out: + out["pinned_panels"] = parse_json_field(out["pinned_panels"]) + if "controlGroupInput" in out and out["controlGroupInput"]: + out["controlGroupInput"] = parse_json_field(out["controlGroupInput"]) + kmeta = out.get("kibanaSavedObjectMeta", {}) + if isinstance(kmeta.get("searchSourceJSON"), str): + out["kibanaSavedObjectMeta"] = { + "searchSourceJSON": json.loads(kmeta["searchSourceJSON"]) + } + return out + + +def to_package_object(live: dict, *, dashboard_id: str, title: str) -> dict: + attrs = normalize_attributes(live["attributes"]) + attrs["title"] = title + return { + "attributes": attrs, + "coreMigrationVersion": live.get("coreMigrationVersion", "8.8.0"), + "id": dashboard_id, + "references": live.get("references", []), + "type": "dashboard", + "typeMigrationVersion": live.get("typeMigrationVersion", "10.3.0"), + } + + +def export_dashboard(dashboard_id: str, title: str) -> Path: + live = kibana_request("GET", f"/api/saved_objects/dashboard/{dashboard_id}") + obj = to_package_object(live, dashboard_id=dashboard_id, title=title) + if dashboard_id in SENTINEL_DASHBOARDS: + obj = apply_sentinel(obj) + path = DASHBOARD_DIR / f"{dashboard_id}.json" + if dashboard_id in SENTINEL_DASHBOARDS: + verify(path, obj) + path.write_text(json.dumps(obj, indent=4) + "\n") + return path + + +def main() -> int: + if not DASHBOARD_DIR.is_dir(): + print(f"Dashboard dir missing: {DASHBOARD_DIR}", file=sys.stderr) + return 1 + + print(f"Exporting Chargeback dashboards from {KIBANA_HOST}") + print(f"Writing to {DASHBOARD_DIR}") + + try: + for dashboard_id, title in DASHBOARDS: + path = export_dashboard(dashboard_id, title) + obj = json.loads(path.read_text()) + pinned = obj["attributes"].get("pinned_panels", {}) + pinned_count = len(pinned.get("panels", {})) if pinned else 0 + links = sum(1 for p in obj["attributes"]["panelsJSON"] if p.get("type") == "links") + print(f" {path.name}: links={links} pinned_controls={pinned_count}") + except urllib.error.HTTPError as exc: + print(f"HTTP {exc.code}: {exc.read().decode()}", file=sys.stderr) + return 1 + + print("Done. Run apply_chargeback_esql_sentinel.py and elastic-package build before release.") + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/scripts/inline_chargeback_links_nav.py b/scripts/inline_chargeback_links_nav.py new file mode 100644 index 0000000..57d9b52 --- /dev/null +++ b/scripts/inline_chargeback_links_nav.py @@ -0,0 +1,178 @@ +#!/usr/bin/env python3 +"""Replace packaged kibana/links savedObjectRef nav with inline links panels. + +package-spec does not allow kibana/links/ (format_version 3.x). Inline links +panels on each dashboard are valid and match the pre-shared-asset shape. +""" +from __future__ import annotations + +import json +import sys +from pathlib import Path + +INTEGRATIONS = Path(__file__).resolve().parents[2] / "integrations" +DASHBOARD_DIR = INTEGRATIONS / "packages/chargeback/kibana/dashboard" +LINKS_DIR = INTEGRATIONS / "packages/chargeback/kibana/links" +NAV_LINKS_ID = "chargeback-e97bc218-1acc-474f-b47c-58d09ce83daf" + +DASHBOARD_IDS = { + "billing": "chargeback-b6cecac0-ebe7-43de-bfc6-1ff287ad860a", + "usage": "chargeback-21588d0e-fb6a-4f76-ad2f-cd7b3d7a3d7c", + "config": "chargeback-7ec9743a-ccfa-43d2-8311-2f25893dadae", +} + +# Stable per-dashboard panel + link entry IDs (from prior inline export). +NAV_SPECS = { + DASHBOARD_IDS["billing"]: { + "panel_id": "c39e2f18-f301-40b0-8f59-2a8547ddffbb", + "link_ids": [ + "cba353e8-3f35-e498-55ec-72e282a480ce", + "4b7dbb50-c024-8666-14ba-f6b4487eced7", + "3a5f90b9-a885-f9a9-6669-49f360b72b47", + ], + }, + DASHBOARD_IDS["usage"]: { + "panel_id": "13cd2a6a-ec89-4553-ae64-cbc347770aa5", + "link_ids": [ + "a11b22c3-d44e-455f-8660-778899001122", + "b22c33d4-e55f-4660-9771-889900112233", + "c33d44e5-f660-4771-a882-990011223344", + ], + }, + DASHBOARD_IDS["config"]: { + "panel_id": "98c09b2b-c68f-ac41-8625-0fc04714f44f", + "link_ids": [ + "cba353e8-3f35-e498-55ec-72e282a480ce", + "4b7dbb50-c024-8666-14ba-f6b4487eced7", + "3a5f90b9-a885-f9a9-6669-49f360b72b47", + ], + }, +} + +LINK_TARGETS = [ + (DASHBOARD_IDS["billing"], "[Chargeback] Billing Components Overview"), + (DASHBOARD_IDS["usage"], "[Chargeback] Usage & Cost Allocation"), + (DASHBOARD_IDS["config"], "[Chargeback] Configuration"), +] + + +def make_inline_nav(panel_id: str, link_ids: list[str]) -> tuple[dict, list[dict]]: + links = [] + refs = [] + for order, (link_id, (dash_id, label)) in enumerate(zip(link_ids, LINK_TARGETS)): + dest = f"link_{link_id}_dashboard" + links.append( + { + "destinationRefName": dest, + "id": link_id, + "label": label, + "options": { + "openInNewTab": False, + "useCurrentDateRange": True, + "useCurrentFilters": True, + }, + "order": order, + "type": "dashboardLink", + } + ) + refs.append( + { + "id": dash_id, + "name": f"{panel_id}:{dest}", + "type": "dashboard", + } + ) + + panel = { + "embeddableConfig": { + "attributes": { + "layout": "horizontal", + "links": links, + }, + "enhancements": {}, + "hide_title": True, + }, + "gridData": {"h": 2, "i": panel_id, "w": 48, "x": 0, "y": 0}, + "panelIndex": panel_id, + "type": "links", + } + return panel, refs + + +def strip_old_nav_refs(references: list[dict], panel_id: str) -> list[dict]: + kept = [] + for ref in references: + name = ref.get("name", "") + if ref.get("type") == "links": + continue + if ref.get("type") == "dashboard" and ( + name.startswith(f"{panel_id}:link_") or ":link_" in name and name.endswith("_dashboard") + ): + # drop old per-panel link refs; re-added below + if name.startswith(f"{panel_id}:"): + continue + if ref.get("id") == NAV_LINKS_ID: + continue + kept.append(ref) + return kept + + +def is_top_nav_panel(panel: dict) -> bool: + grid = panel.get("gridData") or {} + if grid.get("y") != 0 or grid.get("sectionId"): + return False + return panel.get("type") in ("links", "markdown") + + +def apply_dashboard(path: Path, dash_id: str) -> None: + spec = NAV_SPECS[dash_id] + panel_id = spec["panel_id"] + panel, link_refs = make_inline_nav(panel_id, spec["link_ids"]) + + obj = json.loads(path.read_text()) + panels = [p for p in obj["attributes"]["panelsJSON"] if not is_top_nav_panel(p)] + panels.insert(0, panel) + obj["attributes"]["panelsJSON"] = panels + + refs = strip_old_nav_refs(obj.get("references", []), panel_id) + # Also strip any remaining *:savedObjectRef links refs + refs = [r for r in refs if not (r.get("type") == "links" or r.get("name", "").endswith(":savedObjectRef"))] + obj["references"] = link_refs + refs + + path.write_text(json.dumps(obj, indent=4) + "\n") + print(f"OK {path.name}: inline links nav ({panel_id[:8]}…)") + + +def main() -> int: + for dash_id in NAV_SPECS: + apply_dashboard(DASHBOARD_DIR / f"{dash_id}.json", dash_id) + + if LINKS_DIR.is_dir(): + for f in LINKS_DIR.glob("*.json"): + f.unlink() + print(f"Removed {f.relative_to(INTEGRATIONS)}") + try: + LINKS_DIR.rmdir() + print(f"Removed {LINKS_DIR.relative_to(INTEGRATIONS)}/") + except OSError: + pass + + # Verify no packaged links asset and no savedObjectRef to it + assert not LINKS_DIR.exists() or not any(LINKS_DIR.glob("*.json")), LINKS_DIR + for dash_id in NAV_SPECS: + obj = json.loads((DASHBOARD_DIR / f"{dash_id}.json").read_text()) + p0 = obj["attributes"]["panelsJSON"][0] + assert p0["type"] == "links" + assert "attributes" in p0["embeddableConfig"] + assert not any(r.get("type") == "links" for r in obj.get("references", [])) + assert any( + r.get("type") == "dashboard" and ":link_" in r.get("name", "") + for r in obj.get("references", []) + ), dash_id + + print("OK all dashboards use inline links; kibana/links removed") + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/scripts/migrate_chargeback_esql_controls.py b/scripts/migrate_chargeback_esql_controls.py new file mode 100644 index 0000000..7f208e1 --- /dev/null +++ b/scripts/migrate_chargeback_esql_controls.py @@ -0,0 +1,262 @@ +#!/usr/bin/env python3 +"""Migrate Chargeback Billing/Usage dashboards to pinned ES|QL controls. + +Kibana 9.4 does not render links navigation when esql_control panels are embedded in +panelsJSON on the same dashboard. Controls belong in pinned_panels (below nav links). + +- links panel stays in panelsJSON at y=0 +- esql_control panels live in pinned_panels +- control queries are simple option lists (no chained MV_* in control queries) +- selected_options defaults to [__chargeback_unfiltered__] (portable "no filter" sentinel) +- panel ES|QL uses CASE(... sentinel ...) MV_INTERSECTS filter clauses +""" +from __future__ import annotations + +import json +import sys +from copy import deepcopy +from pathlib import Path +from typing import Any + +INTEGRATIONS = Path(__file__).resolve().parents[2] / "integrations" +DASHBOARD_DIR = INTEGRATIONS / "packages/chargeback/kibana/dashboard" + +BILLING_ID = "chargeback-b6cecac0-ebe7-43de-bfc6-1ff287ad860a" +USAGE_ID = "chargeback-21588d0e-fb6a-4f76-ad2f-cd7b3d7a3d7c" + +SENTINEL = "__chargeback_unfiltered__" + +BILLING_CONTROLS = [ + { + "panel_id": "c42ec760-734f-46bb-8e0f-e52f3d4a1f4d", + "order": 0, + "variable_name": "dg_selected", + "title": "Deployment group", + "esql_query": ( + "FROM billing_cluster_cost_lookup\n" + '| WHERE deployment_group != ""\n' + "| STATS COUNT(*) BY deployment_group\n" + "| KEEP deployment_group\n" + "| SORT deployment_group" + ), + }, + { + "panel_id": "5d11a79f-3957-4c0c-a42a-35616191a6fa", + "order": 1, + "variable_name": "deployment_selected", + "title": "Deployment name", + "esql_query": ( + "FROM billing_cluster_cost_lookup\n" + '| WHERE deployment_group != ""\n' + "| STATS COUNT(*) BY deployment_name\n" + "| KEEP deployment_name\n" + "| SORT deployment_name" + ), + }, + { + "panel_id": "2cc4273b-85a3-46d8-aebd-019a383c4b64", + "order": 2, + "variable_name": "cost_category_selected", + "title": "Cost category", + "esql_query": ( + "FROM billing_cluster_cost_lookup\n" + "| WHERE cost_category IS NOT NULL\n" + "| STATS COUNT(*) BY cost_category\n" + "| KEEP cost_category\n" + "| SORT cost_category" + ), + }, + { + "panel_id": "18543502-9e06-4644-9a39-34e39d2eecd4", + "order": 3, + "variable_name": "cost_type_selected", + "title": "Cost type", + "esql_query": ( + "FROM billing_cluster_cost_lookup\n" + "| WHERE cost_type IS NOT NULL\n" + "| STATS COUNT(*) BY cost_type\n" + "| KEEP cost_type\n" + "| SORT cost_type" + ), + }, +] + +USAGE_CONTROLS = [ + { + "panel_id": "b36355dd-4157-4047-9f99-c6979674fd5a", + "order": 0, + "variable_name": "dg_selected", + "title": "Deployment group", + "esql_query": ( + "FROM billing_realized_pool_lookup\n" + '| WHERE deployment_group != ""\n' + "| STATS COUNT(*) BY deployment_group\n" + "| KEEP deployment_group\n" + "| SORT deployment_group" + ), + }, + { + "panel_id": "2900feae-7837-4c04-bb45-5c78ccc46b95", + "order": 1, + "variable_name": "deployment_selected", + "title": "Deployment name", + "esql_query": ( + "FROM billing_realized_pool_lookup\n" + '| WHERE deployment_group != ""\n' + "| STATS COUNT(*) BY deployment_name\n" + "| KEEP deployment_name\n" + "| SORT deployment_name" + ), + }, + { + "panel_id": "ff353537-6404-47e4-8b77-f12580668ae9", + "order": 2, + "variable_name": "tier_selected", + "title": "Data tier", + "esql_query": ( + "FROM cluster_tier_contribution_lookup\n" + '| WHERE tier IS NOT NULL AND tier != "unknown"\n' + "| STATS COUNT(*) BY tier\n" + "| KEEP tier\n" + "| SORT tier" + ), + }, + { + "panel_id": "2e6a43c7-b43b-46dd-b8cd-2da45eced9c7", + "order": 3, + "variable_name": "datastream_selected", + "title": "Data stream", + "esql_query": ( + "FROM cluster_tier_and_datastream_contribution_lookup\n" + "| WHERE datastream IS NOT NULL\n" + "| STATS COUNT(*) BY datastream\n" + "| KEEP datastream\n" + "| SORT datastream" + ), + }, + { + "panel_id": "a1c8e2f4-6b3d-4e9a-9c5f-1d2e3f4a5b6c", + "order": 4, + "variable_name": "ds_type_selected", + "title": "Data stream type", + "esql_query": ( + "FROM cluster_tier_and_datastream_contribution_lookup\n" + "| WHERE ds_type IS NOT NULL\n" + "| STATS COUNT(*) BY ds_type\n" + "| KEEP ds_type\n" + "| SORT ds_type" + ), + }, + { + "panel_id": "b2d9f3e5-7c4e-5f0b-0d6a-2e3f4a5b6c7d", + "order": 5, + "variable_name": "ds_namespace_selected", + "title": "Data stream namespace", + "esql_query": ( + "FROM cluster_tier_and_datastream_contribution_lookup\n" + "| WHERE ds_namespace IS NOT NULL\n" + "| STATS COUNT(*) BY ds_namespace\n" + "| KEEP ds_namespace\n" + "| SORT ds_namespace" + ), + }, +] + + +def make_pinned_panels(specs: list[dict[str, Any]]) -> dict[str, Any]: + panels: dict[str, Any] = {} + for spec in specs: + panels[spec["panel_id"]] = { + "config": { + "control_type": "VALUES_FROM_QUERY", + "esql_query": spec["esql_query"], + "selected_options": [SENTINEL], + "single_select": False, + "title": spec["title"], + "variable_name": spec["variable_name"], + "variable_type": "multi_values", + }, + "grow": False, + "order": spec["order"], + "type": "esql_control", + "width": "medium", + } + return {"panels": panels} + + +def strip_embedded_controls(obj: dict[str, Any], shift_up: int = 2) -> dict[str, Any]: + attrs = obj["attributes"] + had_embedded = any(p.get("type") == "esql_control" for p in attrs["panelsJSON"]) + if not had_embedded: + return obj + + panels = [p for p in attrs["panelsJSON"] if p.get("type") != "esql_control"] + for panel in panels: + grid = panel.get("gridData", {}) + if grid and not grid.get("sectionId") and panel.get("type") != "links": + y = grid.get("y", 0) + if y >= shift_up: + grid["y"] = y - shift_up + for section in attrs.get("sections", []): + grid = section.get("gridData", {}) + y = grid.get("y", 0) + if y >= shift_up: + grid["y"] = y - shift_up + attrs["panelsJSON"] = panels + return obj + + +def migrate_dashboard(obj: dict[str, Any], control_specs: list[dict[str, Any]]) -> dict[str, Any]: + obj = deepcopy(obj) + obj = strip_embedded_controls(obj) + obj["attributes"]["pinned_panels"] = make_pinned_panels(control_specs) + return obj + + +def verify_dashboard(path: Path, control_vars: list[str]) -> None: + obj = json.loads(path.read_text()) + attrs = obj["attributes"] + pinned = attrs.get("pinned_panels", {}).get("panels", {}) + assert pinned, f"{path.name}: pinned_panels required" + + embedded = [p for p in attrs["panelsJSON"] if p.get("type") == "esql_control"] + assert not embedded, f"{path.name}: embedded esql_control breaks nav links" + + seen = [] + for panel in pinned.values(): + cfg = panel["config"] + seen.append(cfg["variable_name"]) + assert cfg["selected_options"] == [SENTINEL], cfg["variable_name"] + assert cfg["esql_query"].strip(), cfg["variable_name"] + assert "MV_CONTAINS" not in cfg["esql_query"], cfg["variable_name"] + assert "MV_INTERSECTS" not in cfg["esql_query"], cfg["variable_name"] + + assert sorted(seen) == sorted(control_vars), (path.name, seen, control_vars) + + text = path.read_text() + assert SENTINEL in text, f"{path.name}: missing sentinel" + assert "MV_CONTAINS" not in text, f"{path.name}: MV_CONTAINS in panel queries" + assert text.count("MV_INTERSECTS") > 0, f"{path.name}: expected MV_INTERSECTS in panel queries" + assert "CASE(?dg_selected" in text and SENTINEL in text, f"{path.name}: sentinel CASE filters" + + +def main() -> int: + billing_path = DASHBOARD_DIR / f"{BILLING_ID}.json" + usage_path = DASHBOARD_DIR / f"{USAGE_ID}.json" + + billing = migrate_dashboard(json.loads(billing_path.read_text()), BILLING_CONTROLS) + usage = migrate_dashboard(json.loads(usage_path.read_text()), USAGE_CONTROLS) + + billing_path.write_text(json.dumps(billing, indent=4) + "\n") + usage_path.write_text(json.dumps(usage, indent=4) + "\n") + + verify_dashboard(billing_path, [c["variable_name"] for c in BILLING_CONTROLS]) + verify_dashboard(usage_path, [c["variable_name"] for c in USAGE_CONTROLS]) + + print(f"Updated {billing_path.name}") + print(f"Updated {usage_path.name}") + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/scripts/reset_chargeback_esql_dashboards.py b/scripts/reset_chargeback_esql_dashboards.py new file mode 100755 index 0000000..ef05ecb --- /dev/null +++ b/scripts/reset_chargeback_esql_dashboards.py @@ -0,0 +1,26 @@ +#!/usr/bin/env python3 +"""Reset Chargeback Billing/Usage dashboards on Kibana from package source. + +Thin wrapper around deploy_chargeback_dashboards_to_kibana.py for the two ES|QL dashboards. +""" +from __future__ import annotations + +import subprocess +import sys +from pathlib import Path + +SCRIPT_DIR = Path(__file__).resolve().parent + + +def main() -> int: + cmd = [ + sys.executable, + str(SCRIPT_DIR / "deploy_chargeback_dashboards_to_kibana.py"), + "--billing", + "--usage", + ] + return subprocess.call(cmd) + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/scripts/restore_chargeback_links_nav.py b/scripts/restore_chargeback_links_nav.py new file mode 100644 index 0000000..d54bcab --- /dev/null +++ b/scripts/restore_chargeback_links_nav.py @@ -0,0 +1,150 @@ +#!/usr/bin/env python3 +"""Remove markdown nav hack; restore native links panels on Billing/Usage. + +Do NOT use add_chargeback_markdown_nav.py — links nav belongs in panelsJSON like Configuration. +""" +from __future__ import annotations + +import json +import subprocess +from copy import deepcopy +from pathlib import Path + +INTEGRATIONS = Path(__file__).resolve().parents[2] / "integrations" +DASHBOARD_DIR = INTEGRATIONS / "packages/chargeback/kibana/dashboard" +REPO = INTEGRATIONS + +DASHBOARDS = ( + "chargeback-b6cecac0-ebe7-43de-bfc6-1ff287ad860a", + "chargeback-21588d0e-fb6a-4f76-ad2f-cd7b3d7a3d7c", +) + +# Simple control queries (no All / MV_APPEND — sentinel lives in selected_options only). +CONTROL_QUERIES = { + "dg_selected": ( + "FROM {index}\n" + '| WHERE deployment_group != ""\n' + "| STATS COUNT(*) BY deployment_group\n" + "| KEEP deployment_group\n" + "| SORT deployment_group" + ), + "deployment_selected": ( + "FROM {index}\n" + '| WHERE deployment_group != ""\n' + "| STATS COUNT(*) BY deployment_name\n" + "| KEEP deployment_name\n" + "| SORT deployment_name" + ), + "cost_category_selected": ( + "FROM billing_cluster_cost_lookup\n" + "| WHERE cost_category IS NOT NULL\n" + "| STATS COUNT(*) BY cost_category\n" + "| KEEP cost_category\n" + "| SORT cost_category" + ), + "cost_type_selected": ( + "FROM billing_cluster_cost_lookup\n" + "| WHERE cost_type IS NOT NULL\n" + "| STATS COUNT(*) BY cost_type\n" + "| KEEP cost_type\n" + "| SORT cost_type" + ), + "tier_selected": ( + "FROM cluster_tier_contribution_lookup\n" + '| WHERE tier IS NOT NULL AND tier != "unknown"\n' + "| STATS COUNT(*) BY tier\n" + "| KEEP tier\n" + "| SORT tier" + ), + "datastream_selected": ( + "FROM cluster_tier_and_datastream_contribution_lookup\n" + "| WHERE datastream IS NOT NULL\n" + "| STATS COUNT(*) BY datastream\n" + "| KEEP datastream\n" + "| SORT datastream" + ), + "ds_type_selected": ( + "FROM cluster_tier_and_datastream_contribution_lookup\n" + "| WHERE ds_type IS NOT NULL\n" + "| STATS COUNT(*) BY ds_type\n" + "| KEEP ds_type\n" + "| SORT ds_type" + ), + "ds_namespace_selected": ( + "FROM cluster_tier_and_datastream_contribution_lookup\n" + "| WHERE ds_namespace IS NOT NULL\n" + "| STATS COUNT(*) BY ds_namespace\n" + "| KEEP ds_namespace\n" + "| SORT ds_namespace" + ), +} + +BILLING_INDEX = "billing_cluster_cost_lookup" +USAGE_INDEX = "billing_realized_pool_lookup" + + +def git_head_dashboard(dash_id: str) -> dict: + raw = subprocess.check_output( + ["git", "-C", str(REPO), "show", f"HEAD:packages/chargeback/kibana/dashboard/{dash_id}.json"], + text=True, + ) + return json.loads(raw) + + +def links_nav_from_git(dash_id: str) -> tuple[dict, list]: + obj = git_head_dashboard(dash_id) + panels = obj["attributes"]["panelsJSON"] + links = next(p for p in panels if p.get("type") == "links") + refs = [r for r in obj.get("references", []) if ":link_" in r.get("name", "")] + return deepcopy(links), deepcopy(refs) + + +def restore_dashboard(path: Path, dash_id: str, index: str) -> None: + obj = json.loads(path.read_text()) + panels = obj["attributes"]["panelsJSON"] + links_panel, link_refs = links_nav_from_git(dash_id) + + panels = [p for p in panels if not (p.get("type") == "markdown" and p.get("gridData", {}).get("y") == 0)] + if not any(p.get("type") == "links" for p in panels): + panels.insert(0, links_panel) + + for panel in panels: + if panel.get("type") != "esql_control": + continue + cfg = panel["embeddableConfig"] + var = cfg.get("variable_name", "") + if var not in CONTROL_QUERIES: + continue + q = CONTROL_QUERIES[var] + if "{index}" in q: + q = q.format(index=index) + cfg["esql_query"] = q + + obj["attributes"]["panelsJSON"] = panels + other_refs = [r for r in obj.get("references", []) if ":link_" not in r.get("name", "")] + obj["references"] = link_refs + other_refs + + path.write_text(json.dumps(obj, indent=4) + "\n") + print(f"OK {path.name}: links nav restored, markdown removed") + + +def main() -> int: + restore_dashboard(DASHBOARD_DIR / f"{DASHBOARDS[0]}.json", DASHBOARDS[0], BILLING_INDEX) + restore_dashboard(DASHBOARD_DIR / f"{DASHBOARDS[1]}.json", DASHBOARDS[1], USAGE_INDEX) + + import sys + + sys.path.insert(0, str(Path(__file__).resolve().parent)) + from apply_chargeback_esql_sentinel import apply_sentinel, verify + + for dash_id in DASHBOARDS: + path = DASHBOARD_DIR / f"{dash_id}.json" + obj = apply_sentinel(json.loads(path.read_text())) + verify(path, obj) + path.write_text(json.dumps(obj, indent=4) + "\n") + print(f"OK {path.name}: sentinel verified") + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/scripts/run_e2e_tests.sh b/scripts/run_e2e_tests.sh index d5a8017..a1fd358 100755 --- a/scripts/run_e2e_tests.sh +++ b/scripts/run_e2e_tests.sh @@ -153,11 +153,11 @@ if [[ "${MONITORING_COUNT_EARLY:-0}" -lt 810 ]]; then prod) b_idx=10000; b_qry=5000; b_store=53687091200 ;; monitoring) b_idx=2000; b_qry=800; b_store=5368709120 ;; esac - for ds in "logs-app" "metrics-system" "traces-apm"; do + for ds in "logs-app-team_a" "metrics-system-team_b" "custom_index"; do case "$ds" in - logs-app) dsf=10 ;; - metrics-system) dsf=6 ;; - traces-apm) dsf=14 ;; + logs-app-team_a) dsf=10 ;; + metrics-system-team_b) dsf=6 ;; + custom_index) dsf=14 ;; esac for tier_pref in "data_hot,data_content" "data_warm" "data_cold"; do case "$tier_pref" in @@ -262,11 +262,29 @@ fi # 7. Build and install Chargeback echo "--- 7. Build and install Chargeback ---" if [[ "$REPLACE_CHARGEBACK_DASHBOARD" == "1" ]]; then - echo "Replacing Chargeback dashboard/index pattern from local package (REPLACE_CHARGEBACK_DASHBOARD=1)." - curl_kibana -X DELETE "$KIBANA_HOST/api/saved_objects/dashboard/chargeback-39a39857-746c-4a29-adca-3c2fcb6bcfb6" >/dev/null 2>&1 && echo " Deleted Chargeback dashboard." || true + echo "Replacing Chargeback dashboards/index pattern from local package (REPLACE_CHARGEBACK_DASHBOARD=1)." + for DASH_ID in \ + chargeback-39a39857-746c-4a29-adca-3c2fcb6bcfb6 \ + chargeback-b6cecac0-ebe7-43de-bfc6-1ff287ad860a \ + chargeback-21588d0e-fb6a-4f76-ad2f-cd7b3d7a3d7c; do + curl_kibana -X DELETE "$KIBANA_HOST/api/saved_objects/dashboard/$DASH_ID" >/dev/null 2>&1 && echo " Deleted dashboard $DASH_ID." || true + done curl_kibana -X DELETE "$KIBANA_HOST/api/saved_objects/index-pattern/chargeback_integration" >/dev/null 2>&1 && echo " Deleted Chargeback index pattern." || true fi -(cd "$INTEGRATIONS_REPO/packages/chargeback" && $EP_CMD build --skip-validation && $EP_CMD install --skip-validation) +CHARGEBACK_INSTALL_ZIP="${CHARGEBACK_INSTALL_ZIP:-$CHARGEBACK_REPO/integration/assets/0.5.1/chargeback-0.5.1.zip}" +if [[ -f "$CHARGEBACK_INSTALL_ZIP" ]]; then + echo " Installing from chargeback repo zip: $CHARGEBACK_INSTALL_ZIP" + (cd "$INTEGRATIONS_REPO" && $EP_CMD install --zip "$CHARGEBACK_INSTALL_ZIP" --skip-validation) +else + echo " No zip at $CHARGEBACK_INSTALL_ZIP; building from integrations packages/chargeback" + (cd "$INTEGRATIONS_REPO/packages/chargeback" && $EP_CMD build --skip-validation && $EP_CMD install --skip-validation) +fi + +# 7a. Ensure Billing/Usage ES|QL control queries match package source (pinned_panels) +if [[ -x "$CHARGEBACK_REPO/scripts/reset_chargeback_esql_dashboards.py" ]]; then + echo "--- 7a. Sync Billing/Usage ES|QL control queries from package ---" + INTEGRATIONS_REPO="$INTEGRATIONS_REPO" python3 "$CHARGEBACK_REPO/scripts/reset_chargeback_esql_dashboards.py" || exit 1 +fi # 7b. Start Chargeback transforms (package has start: true but installer may not start them) echo "--- 7b. Start Chargeback transforms ---" @@ -467,7 +485,7 @@ echo "--- 11. Cross-verification: data match across all *lookup indices ---" CROSS_OK=1 if command -v jq >/dev/null 2>&1; then # Key fields to compare across indices (order for table). billing_cluster_cost_lookup uses total_chargeable_units (Chargeback 0.3.0). - KEY_FIELDS="deployment_id @timestamp deployment_name total_chargeable_units tier datastream composite_key cost_type" + KEY_FIELDS="deployment_id @timestamp deployment_name total_chargeable_units tier datastream ds_type ds_namespace composite_key cost_type" # Build table: rows = field names, columns = lookup indices (same row = same field across indices) PASTE_TMP=$(mktemp) trap 'rm -f "$PASTE_TMP"' EXIT diff --git a/scripts/seed_e2e_dataset.sh b/scripts/seed_e2e_dataset.sh new file mode 100755 index 0000000..c0a9205 --- /dev/null +++ b/scripts/seed_e2e_dataset.sh @@ -0,0 +1,483 @@ +#!/usr/bin/env bash +# Diverse Chargeback E2E dataset: monitoring-indices, on-prem/cloud billing, node_stats. +# Source from run_e2e_tests.sh or run standalone against an existing stack with Chargeback installed. +# +# Usage: +# ./scripts/seed_e2e_dataset.sh # seed + reset Chargeback/on-prem transforms +# ./scripts/seed_e2e_dataset.sh --seed-only # seed sources only (no transform reset) +# +# Env: E2E_DAYS (default 21), ES_HOST, ELASTIC_USER, ELASTIC_PASSWORD, SKUTEST_DEPLOYMENT (default skutest) +if [[ "${BASH_SOURCE[0]}" == "${0}" ]]; then + set -euo pipefail +fi + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +ES_HOST="${ES_HOST:-https://127.0.0.1:9200}" +USER="${ELASTIC_USER:-elastic}" +PASS="${ELASTIC_PASSWORD:-changeme}" +E2E_DAYS="${E2E_DAYS:-21}" +SKUTEST_DEPLOYMENT="${SKUTEST_DEPLOYMENT:-skutest}" + +# shellcheck source=scripts/e2e_lib.sh +source "$SCRIPT_DIR/e2e_lib.sh" + +# On-prem: id|chargeback_group|erus|base_ecu|idx_base|qry_base|store_gb|heap_pct|disk_pct|workload_profile +E2E_ONPREM_DEPLOYMENTS=( + "dev|product|2|2000|3000|1500|10|45|40|balanced" + "staging|product|3|3500|5000|2200|20|62|55|balanced" + "prod|product|10|10000|12000|6000|50|82|72|balanced" + "monitoring|monitoring|2|2000|2000|800|5|91|88|metrics_heavy" + "analytics|research|4|4500|4000|9000|35|55|35|query_heavy" + "security|platform|3|3200|2500|1200|25|70|65|logs_heavy" +) + +# Data streams: name|index_weight|query_weight|store_weight (relative 1-20) +# Prefer type-dataset-namespace names; include one non-conforming name → ds_* = other +E2E_DATASTREAMS=( + "logs-app-team_a|12|8|10" + "logs-security-team_b|10|6|9" + "metrics-system-team_a|8|5|7" + "metrics-infra-team_b|7|4|8" + "traces-apm-team_a|14|12|6" + "custom_index|5|9|4" +) + +# Tiers: preference|index_mult|query_mult|store_mult +E2E_TIERS=( + "data_hot,data_content|10|10|1" + "data_warm|3|2|3" + "data_cold|1|1|10" + "data_frozen|1|1|18" +) + +# Cloud fixture for issue #8 (single recent day, multi-SKU) +E2E_SKUTEST_SKUS=( + "aws.es.datahot.general-purpose:1200" + "aws.es.ml.general-purpose:400" + "aws.kibana.general-purpose:300" +) + +# Cloud-prod: daily SKUs with tier + platform + transfer/snapshot variety +E2E_CLOUD_PROD_SKUS=( + "aws.es.datahot.general-purpose" + "aws.es.datacontent.general-purpose" + "aws.es.datawarm.general-purpose" + "aws.es.datacold.general-purpose" + "aws.es.ml.general-purpose" + "aws.kibana.general-purpose" + "aws.data-transfer.inter-region" + "aws.snapshot-storage.standard" +) + +e2e_curl_es() { + curl -sS -k -u "$USER:$PASS" -H "Content-Type: application/json" "$@" +} + +e2e_timestamp_days_ago() { + local day=$1 + date -u -v-"${day}"d 2>/dev/null +%Y-%m-%dT00:00:00.000Z \ + || date -u -d "$day days ago" 2>/dev/null +%Y-%m-%dT00:00:00.000Z +} + +e2e_dep_field() { + local dep=$1 field=$2 + local row + for row in "${E2E_ONPREM_DEPLOYMENTS[@]}"; do + if [[ "${row%%|*}" == "$dep" ]]; then + local rest=${row#*|} + local i=1 + for f in group erus base_ecu idx_base qry_base store_gb heap_pct disk_pct profile; do + if [[ "$f" == "$field" ]]; then + echo "${rest%%|*}" + return 0 + fi + rest=${rest#*|} + done + fi + done + return 1 +} + +e2e_onprem_deployment_ids() { + local row + for row in "${E2E_ONPREM_DEPLOYMENTS[@]}"; do + echo "${row%%|*}" + done +} + +e2e_profile_ds_multiplier() { + local profile=$1 ds=$2 kind=$3 + local _ds iw qw sw + for _ds in "${E2E_DATASTREAMS[@]}"; do + [[ "${_ds%%|*}" == "$ds" ]] || continue + iw=${_ds#*|}; qw=${iw#*|}; sw=${qw#*|} + iw=${iw%%|*}; qw=${qw%%|*}; sw=${sw%%|*} + case "$profile:$kind" in + balanced:index) echo "$iw" ;; + balanced:query) echo "$qw" ;; + balanced:store) echo "$sw" ;; + logs_heavy:index) + case "$ds" in logs-app-team_a|logs-security-team_b) echo $(( iw * 15 / 10 )) ;; *) echo $(( iw * 7 / 10 )) ;; esac ;; + logs_heavy:query) + case "$ds" in logs-app-team_a|logs-security-team_b) echo $(( qw * 12 / 10 )) ;; *) echo $(( qw * 8 / 10 )) ;; esac ;; + logs_heavy:store) + case "$ds" in logs-app-team_a|logs-security-team_b) echo $(( sw * 14 / 10 )) ;; *) echo $(( sw * 6 / 10 )) ;; esac ;; + query_heavy:index) echo $(( iw * 8 / 10 )) ;; + query_heavy:query) + case "$ds" in traces-apm-team_a|custom_index) echo $(( qw * 18 / 10 )) ;; *) echo $(( qw * 9 / 10 )) ;; esac ;; + query_heavy:store) echo $(( sw * 7 / 10 )) ;; + metrics_heavy:index) + case "$ds" in metrics-system-team_a|metrics-infra-team_b) echo $(( iw * 16 / 10 )) ;; *) echo $(( iw * 6 / 10 )) ;; esac ;; + metrics_heavy:query) + case "$ds" in metrics-system-team_a|metrics-infra-team_b) echo $(( qw * 14 / 10 )) ;; *) echo $(( qw * 7 / 10 )) ;; esac ;; + metrics_heavy:store) + case "$ds" in metrics-system-team_a|metrics-infra-team_b) echo $(( sw * 12 / 10 )) ;; *) echo $(( sw * 8 / 10 )) ;; esac ;; + *) echo 10 ;; + esac + return 0 + done + echo 10 +} + +e2e_compute_monitoring_expected() { + local n_clusters=${#E2E_ONPREM_DEPLOYMENTS[@]} + local n_ds=${#E2E_DATASTREAMS[@]} + local n_tiers=${#E2E_TIERS[@]} + echo $(( E2E_DAYS * n_clusters * n_ds * n_tiers )) +} + +e2e_seed_monitoring_indices() { + local expected force=${SEED_FORCE:-0} + expected=$(e2e_compute_monitoring_expected) + MONITORING_EXPECTED=$expected + export MONITORING_EXPECTED + + local count + count=$(e2e_es_count monitoring-indices) + if [[ "${count:-0}" -ge "$expected" && "$force" != "1" ]]; then + echo "monitoring-indices: ${count} docs (target ${expected}); skip seed (set SEED_FORCE=1 to replace)." + return 0 + fi + + echo "Seeding monitoring-indices: ${E2E_DAYS}d × ${#E2E_ONPREM_DEPLOYMENTS[@]} clusters × ${#E2E_DATASTREAMS[@]} data streams × ${#E2E_TIERS[@]} tiers (${expected} docs)." + [[ "${count:-0}" -gt 0 ]] && e2e_curl_es -X DELETE "$ES_HOST/monitoring-indices" >/dev/null 2>&1 || true + + local bulk day ts cluster row cluster_id group _erus base_ecu idx_base qry_base store_gb heap disk profile + bulk=$(mktemp) + for day in $(seq 1 "$E2E_DAYS"); do + TS=$(e2e_timestamp_days_ago "$day") + # Weekend dip + slow upward trend + local day_mod=$(( day % 7 )) + local trend=$(( 92 + day % 15 )) + local weekend=100 + [[ "$day_mod" -eq 0 || "$day_mod" -eq 6 ]] && weekend=72 + + for row in "${E2E_ONPREM_DEPLOYMENTS[@]}"; do + IFS='|' read -r cluster_id group _erus base_ecu idx_base qry_base store_gb heap disk profile <<< "$row" + local store_bytes=$(( store_gb * 1073741824 )) + for ds_row in "${E2E_DATASTREAMS[@]}"; do + local ds=${ds_row%%|*} + local ds_i ds_q ds_s tier_row tier_pref t_i t_q t_s + ds_i=$(e2e_profile_ds_multiplier "$profile" "$ds" index) + ds_q=$(e2e_profile_ds_multiplier "$profile" "$ds" query) + ds_s=$(e2e_profile_ds_multiplier "$profile" "$ds" store) + for tier_row in "${E2E_TIERS[@]}"; do + IFS='|' read -r tier_pref t_i t_q t_s <<< "$tier_row" + local v=$(( trend * weekend / 100 )) + local idx=$(( idx_base * ds_i / 10 * t_i / 10 * v / 100 )) + local qry=$(( qry_base * ds_q / 10 * t_q / 10 * v / 100 )) + local sto=$(( store_bytes * ds_s / 10 * t_s / 10 )) + printf '{"index":{"_index":"monitoring-indices"}}\n' >> "$bulk" + printf '{"@timestamp":"%s","elasticsearch":{"cluster":{"name":"%s"},"index":{"datastream":"%s","tier_preference":"%s","total":{"indexing":{"index_time_in_millis":%d},"search":{"query_time_in_millis":%d},"store":{"size_in_bytes":%d}},"primaries":{"store":{"total_data_set_size_in_bytes":%d}}}}}\n' \ + "$TS" "$cluster_id" "$ds" "$tier_pref" "$idx" "$qry" "$sto" "$sto" >> "$bulk" + done + done + done + done + e2e_bulk_ndjson "$bulk" + rm -f "$bulk" + echo "Seeded monitoring-indices (${expected} docs)." +} + +e2e_configure_onprem_org() { + local total_erus=0 row erus + for row in "${E2E_ONPREM_DEPLOYMENTS[@]}"; do + erus=$(echo "$row" | cut -d'|' -f3) + total_erus=$(( total_erus + erus )) + done + local annual_cost=$(( total_erus * 15000 )) + echo "Configure onprem_billing_config: ${total_erus} ERU licence, ${#E2E_ONPREM_DEPLOYMENTS[@]} deployments." + e2e_curl_es -X PUT "$ES_HOST/onprem_billing_config/_doc/organization" \ + -d "{\"config_type\":\"organization\",\"total_annual_license_cost\":${annual_cost},\"total_erus_purchased\":${total_erus},\"eru_to_ram_gb\":64,\"currency_unit\":\"EUR\"}" \ + >/dev/null 2>&1 || true + + local dep_id group name tags meru doc_id + for row in "${E2E_ONPREM_DEPLOYMENTS[@]}"; do + IFS='|' read -r dep_id group erus _ _ _ _ _ _ _ <<< "$row" + name="$dep_id" + tags="[\"chargeback_group:${group}\"]" + meru=$(( erus * 1000 )) + doc_id=$(e2e_curl_es "$ES_HOST/onprem_billing_config/_search?size=1" \ + -d "{\"_source\":false,\"query\":{\"term\":{\"deployment_id\":\"$dep_id\"}}}" 2>/dev/null \ + | jq -r '.hits.hits[0]._id // empty') + if [[ -n "$doc_id" ]]; then + e2e_curl_es -X POST "$ES_HOST/onprem_billing_config/_update/$doc_id" \ + -d "{\"doc\":{\"deployment_name\":\"$name\",\"deployment_tags\":$tags,\"deployment_erus\":$erus,\"daily_meru\":$meru}}" \ + >/dev/null 2>&1 || true + else + e2e_curl_es -X PUT "$ES_HOST/onprem_billing_config/_doc/$dep_id" \ + -d "{\"deployment_id\":\"$dep_id\",\"deployment_name\":\"$name\",\"deployment_tags\":$tags,\"deployment_erus\":$erus,\"daily_meru\":$meru}" \ + >/dev/null 2>&1 || true + fi + done +} + +# Re-discover deployments from monitoring-indices, then apply the 24 ERU org + per-deployment config. +e2e_refresh_onprem_config() { + local tid + tid=$(e2e_get_transform_id '^logs-onprem_billing\.config_bootstrap-') + if [[ -z "$tid" ]]; then + echo " WARN: onprem_billing config_bootstrap not installed; configuring org/deployment docs only." + e2e_configure_onprem_org + return 0 + fi + echo "Refresh onprem_billing_config from monitoring-indices (config_bootstrap)..." + e2e_transform_stop "$tid" 1 || true + e2e_transform_reset "$tid" || true + e2e_transform_start "$tid" || true + e2e_wait_for_onprem_config || echo " WARN: onprem_billing_config not fully populated (config_bootstrap may still be running)." + e2e_configure_onprem_org +} + +e2e_seed_billing_onprem() { + local index="metrics-ess_billing.billing-onprem" + echo "Seeding $index: ${E2E_DAYS}d × ${#E2E_ONPREM_DEPLOYMENTS[@]} on-prem deployments." + e2e_curl_es -X POST "$ES_HOST/$index/_delete_by_query?refresh=true&conflicts=proceed" \ + -d '{"query":{"match_all":{}}}' >/dev/null 2>&1 || true + + local bulk now_iso day ts row dep_id group base_ecu name tags ecu v + bulk=$(mktemp) + now_iso=$(date -u 2>/dev/null +%Y-%m-%dT%H:%M:%S.000Z || echo "2026-01-01T00:00:00.000Z") + for day in $(seq 1 "$E2E_DAYS"); do + ts=$(e2e_timestamp_days_ago "$day") + for row in "${E2E_ONPREM_DEPLOYMENTS[@]}"; do + IFS='|' read -r dep_id group _erus base_ecu _ _ _ _ _ _ <<< "$row" + name="$dep_id" + tags="[\"chargeback_group:${group}\"]" + # Per-deployment phase offset so lines diverge on charts + v=$(( 88 + (day + ${#dep_id}) % 24 )) + ecu=$(( base_ecu * v / 100 )) + printf '{"create":{"_index":"%s"}}\n' "$index" >> "$bulk" + printf '{"@timestamp":"%s","event":{"ingested":"%s"},"ess":{"billing":{"deployment_id":"%s","deployment_name":"%s","total_ecu":%d,"deployment_tags":%s,"sku":"onprem_node","type":"capacity","kind":"elasticsearch","deployment_type":"onprem"}}}\n' \ + "$ts" "$now_iso" "$dep_id" "$name" "$ecu" "$tags" >> "$bulk" + done + done + e2e_bulk_ndjson "$bulk" + rm -f "$bulk" + echo "Seeded $index." +} + +e2e_seed_billing_cloud() { + local cloud_index="metrics-ess_billing.billing-default" + local now_iso ts_recent ts_day sku_line sku ecu base i dep_id + + now_iso=$(date -u 2>/dev/null +%Y-%m-%dT%H:%M:%S.000Z || echo "2026-01-01T00:00:00.000Z") + ts_recent=$(date -u -v-3M 2>/dev/null +%Y-%m-%dT%H:%M:%S.000Z || date -u -d '3 minutes ago' 2>/dev/null +%Y-%m-%dT%H:%M:%S.000Z || e2e_timestamp_days_ago 1) + + echo "Seeding $cloud_index: skutest (#8 fixture) + cloud-prod (multi-SKU history)." + e2e_curl_es -X POST "$ES_HOST/$cloud_index/_delete_by_query?refresh=true&conflicts=proceed" \ + -d '{"query":{"terms":{"ess.billing.deployment_id":["'"$SKUTEST_DEPLOYMENT"'","cloud-prod"]}}}' \ + >/dev/null 2>&1 || true + + # skutest: fixed recent snapshot for issue #8 allocatable-ECU check (datahot only = 1200) + for sku_line in "${E2E_SKUTEST_SKUS[@]}"; do + sku=${sku_line%%:*} + ecu=${sku_line##*:} + e2e_curl_es -X POST "$ES_HOST/$cloud_index/_doc?refresh=true" -d "{ + \"@timestamp\": \"$ts_recent\", + \"event\": {\"ingested\": \"$now_iso\"}, + \"ess\": {\"billing\": { + \"deployment_id\": \"$SKUTEST_DEPLOYMENT\", + \"deployment_name\": \"$SKUTEST_DEPLOYMENT\", + \"deployment_tags\": [\"chargeback_group:product\"], + \"total_ecu\": $ecu, + \"sku\": \"$sku\", + \"type\": \"capacity\", + \"kind\": \"elasticsearch\", + \"deployment_type\": \"deployment\" + }} + }" >/dev/null 2>&1 || true + done + + # skutest + cloud-prod: daily history with tier/platform/transfer SKUs + local bulk + bulk=$(mktemp) + for day in $(seq 1 "$E2E_DAYS"); do + ts_day=$(e2e_timestamp_days_ago "$day") + local v=$(( 85 + day % 25 )) + for dep_id in "$SKUTEST_DEPLOYMENT" cloud-prod; do + local tags='["chargeback_group:product"]' + [[ "$dep_id" == "cloud-prod" ]] && tags='["chargeback_group:product","env:production"]' + i=0 + for sku in "${E2E_CLOUD_PROD_SKUS[@]}"; do + i=$(( i + 1 )) + case "$sku" in + *datahot*) base=1400 ;; + *datacontent*) base=600 ;; + *datawarm*) base=900 ;; + *datacold*) base=400 ;; + *datafrozen*) base=200 ;; + *ml*) base=500 ;; + *kibana*) base=350 ;; + *data-transfer*) base=80 ;; + *snapshot*) base=45 ;; + *) base=300 ;; + esac + # skutest history uses smaller amounts; recent fixture above carries the #8 proof + [[ "$dep_id" == "$SKUTEST_DEPLOYMENT" && "$day" -gt 3 ]] && continue + ecu=$(( base * v / 100 * (10 + i % 4) / 10 )) + local kind="elasticsearch" btype="capacity" + [[ "$sku" == *kibana* ]] && kind="kibana" + [[ "$sku" == *data-transfer* || "$sku" == *snapshot* ]] && btype="usage" + printf '{"create":{"_index":"%s"}}\n' "$cloud_index" >> "$bulk" + printf '{"@timestamp":"%s","event":{"ingested":"%s"},"ess":{"billing":{"deployment_id":"%s","deployment_name":"%s","deployment_tags":%s,"total_ecu":%d,"sku":"%s","type":"%s","kind":"%s","deployment_type":"deployment"}}}\n' \ + "$ts_day" "$now_iso" "$dep_id" "$dep_id" "$tags" "$ecu" "$sku" "$btype" "$kind" >> "$bulk" + done + done + done + e2e_bulk_ndjson "$bulk" + rm -f "$bulk" + echo "Seeded $cloud_index (skutest + cloud-prod)." +} + +e2e_seed_node_stats() { + local index="metrics-elasticsearch.stack_monitoring.node_stats-default" + local clusters=() row dep_id + for row in "${E2E_ONPREM_DEPLOYMENTS[@]}"; do + clusters+=("${row%%|*}") + done + clusters+=("$SKUTEST_DEPLOYMENT" "cloud-prod") + + # Metrics data streams only accept docs in the current backing index time range (not 21d history). + # Seed recent samples so cluster_capacity_utilization can compute p95 per deployment for today. + echo "Seeding $index: ${#clusters[@]} clusters × 3 nodes (recent timestamps, variable utilization)." + local cluster_list + cluster_list=$(printf '"%s",' "${clusters[@]}") + cluster_list="[${cluster_list%,}]" + e2e_curl_es -X POST "$ES_HOST/$index/_delete_by_query?refresh=true&conflicts=proceed" \ + -d "{\"query\":{\"terms\":{\"elasticsearch.cluster.name\":$cluster_list}}}" \ + >/dev/null 2>&1 || true + + local cluster_i ts ingested_ts cluster heap disk node_i heap_v disk_v total avail node_name + ingested_ts=$(date -u +%Y-%m-%dT%H:%M:%S.000Z) + cluster_i=0 + for cluster in "${clusters[@]}"; do + cluster_i=$(( cluster_i + 1 )) + if heap=$(e2e_dep_field "$cluster" heap_pct 2>/dev/null); then + disk=$(e2e_dep_field "$cluster" disk_pct) + else + case "$cluster" in + "$SKUTEST_DEPLOYMENT") heap=78; disk=60 ;; + cloud-prod) heap=84; disk=70 ;; + *) heap=75; disk=60 ;; + esac + fi + # Stagger timestamps within the writable window (minutes ago per cluster) + ts=$(date -u -v-$(( cluster_i * 8 + 5 ))M 2>/dev/null +%Y-%m-%dT%H:%M:%S.000Z \ + || date -u -d "$(( cluster_i * 8 + 5 )) minutes ago" 2>/dev/null +%Y-%m-%dT%H:%M:%S.000Z) + for node_i in 1 2 3; do + heap_v=$(( heap + node_i * 2 - 3 )) + disk_v=$(( disk + node_i - 2 )) + [[ "$heap_v" -lt 5 ]] && heap_v=5 + [[ "$heap_v" -gt 99 ]] && heap_v=99 + [[ "$disk_v" -lt 5 ]] && disk_v=5 + [[ "$disk_v" -gt 99 ]] && disk_v=99 + total=1000000 + avail=$(( total * (100 - disk_v) / 100 )) + local node_name="e2e-${cluster}-node-${node_i}" + e2e_curl_es -X POST "$ES_HOST/$index/_doc?refresh=false" -d "{ + \"@timestamp\": \"$ts\", + \"event\": {\"ingested\": \"$ingested_ts\"}, + \"agent\": {\"id\": \"e2e-agent-${cluster}\"}, + \"host\": {\"name\": \"e2e-${cluster}\"}, + \"service\": {\"address\": \"https://${cluster}:9200\"}, + \"elasticsearch\": { + \"cluster\": {\"name\": \"$cluster\", \"id\": \"e2e-cluster-${cluster}\"}, + \"node\": { + \"id\": \"$node_name\", + \"name\": \"$node_name\", + \"roles\": [\"data_hot\", \"data_content\"], + \"stats\": { + \"jvm\": {\"mem\": {\"heap\": {\"used\": {\"pct\": $heap_v}}}}, + \"fs\": {\"total\": {\"total_in_bytes\": $total, \"available_in_bytes\": $avail}} + } + } + } + }" >/dev/null 2>&1 || true + done + done + e2e_curl_es -X POST "$ES_HOST/$index/_refresh" >/dev/null 2>&1 || true + echo "Seeded $index." +} + +e2e_seed_all_sources() { + e2e_seed_monitoring_indices + e2e_seed_billing_onprem + e2e_seed_billing_cloud + e2e_seed_node_stats +} + +e2e_print_dataset_summary() { + echo "" + echo "=== Diverse E2E dataset summary ===" + echo "Horizon: ${E2E_DAYS} days" + echo "On-prem: $(e2e_onprem_deployment_ids | tr '\n' ' ')" + echo "Groups: product, monitoring, research, platform" + echo "Data streams: $(printf '%s ' "${E2E_DATASTREAMS[@]%%|*}")" + echo "Tiers: $(printf '%s ' "${E2E_TIERS[@]%%|*}")" + echo "Cloud: ${SKUTEST_DEPLOYMENT} (issue #8), cloud-prod (multi-SKU + transfer/snapshot)" + echo "Monitoring docs target: $(e2e_compute_monitoring_expected)" + echo "Dashboard: last 30 days recommended" + echo "" +} + +seed_e2e_main() { + local reset=1 + while [[ $# -gt 0 ]]; do + case "$1" in + --seed-only) reset=0; shift ;; + -h|--help) + sed -n '1,12p' "$0" + exit 0 + ;; + *) echo "Unknown option: $1" >&2; exit 1 ;; + esac + done + + e2e_require_commands jq curl + + # Allow parent to supply curl_es + if declare -F curl_es >/dev/null 2>&1; then + e2e_curl_es() { curl_es "$@"; } + fi + + e2e_print_dataset_summary + e2e_seed_monitoring_indices + e2e_refresh_onprem_config + e2e_ensure_onprem_billing_event_ingested_mapping + e2e_seed_billing_onprem + e2e_seed_billing_cloud + e2e_seed_node_stats + if [[ "$reset" -eq 1 ]]; then + e2e_prepare_all_chargeback_transforms + e2e_reset_billing_transforms + e2e_wait_for_contribution_lookups + e2e_wait_for_lookup_indices "$E2E_LOOKUP_MIN_DOCS" + e2e_assert_deployment_groups + fi + echo "Seed complete." +} + +if [[ "${BASH_SOURCE[0]}" == "${0}" ]]; then + seed_e2e_main "$@" +fi diff --git a/scripts/sync_chargeback_from_integrations.sh b/scripts/sync_chargeback_from_integrations.sh index bc556c4..bc75731 100755 --- a/scripts/sync_chargeback_from_integrations.sh +++ b/scripts/sync_chargeback_from_integrations.sh @@ -59,7 +59,8 @@ echo "Destination: $DEST_ZIP" if [[ "$SKIP_BUILD" -eq 0 ]]; then echo "--- elastic-package build ---" - (cd "$INTEGRATIONS_REPO/packages/chargeback" && $EP_CMD build) + # Chargeback ships kibana/links nav panels; package validation rejects them (acknowledged). + (cd "$INTEGRATIONS_REPO/packages/chargeback" && $EP_CMD build --skip-validation) else echo "--- skip build (--skip-build) ---" fi diff --git a/scripts/verify_chargeback_dashboard_package.py b/scripts/verify_chargeback_dashboard_package.py new file mode 100644 index 0000000..8960542 --- /dev/null +++ b/scripts/verify_chargeback_dashboard_package.py @@ -0,0 +1,102 @@ +#!/usr/bin/env python3 +"""Verify Chargeback integration dashboard JSON (and optional built zip) ship sentinel + nav. + +Exit 0 only when Billing/Usage dashboards have: +- links nav in panelsJSON at y=0 (no markdown nav) +- embedded esql_control panels with selected_options: ["__chargeback_unfiltered__"] +- CASE/MV_INTERSECTS sentinel filters in panel ES|QL +- no empty selected_options or MV_CONTAINS + +Usage: + python3 scripts/verify_chargeback_dashboard_package.py + python3 scripts/verify_chargeback_dashboard_package.py --zip path/to/chargeback-0.5.0.zip +""" +from __future__ import annotations + +import argparse +import json +import sys +import zipfile +from pathlib import Path + +SCRIPT_DIR = Path(__file__).resolve().parent +sys.path.insert(0, str(SCRIPT_DIR)) +from apply_chargeback_esql_sentinel import DASHBOARDS, SENTINEL, verify # noqa: E402 + +INTEGRATIONS_ROOT = Path(__file__).resolve().parents[2] / "integrations" +DASHBOARD_DIR = INTEGRATIONS_ROOT / "packages/chargeback/kibana/dashboard" +CONFIG_ID = "chargeback-7ec9743a-ccfa-43d2-8311-2f25893dadae" + + +def verify_config_obj(obj: dict, label: str) -> None: + panels = obj["attributes"]["panelsJSON"] + if isinstance(panels, str): + panels = json.loads(panels) + assert any(p.get("type") == "links" for p in panels), f"{label}: missing links nav" + assert not obj["attributes"].get("pinned_panels"), f"{label}: unexpected pinned_panels" + + +def verify_config(path: Path) -> None: + verify_config_obj(json.loads(path.read_text()), path.name) + + +def verify_source_tree() -> None: + # package-spec does not allow kibana/links/; nav must be inline on each dashboard + links_dir = DASHBOARD_DIR.parent / "links" + assert not links_dir.exists() or not any(links_dir.glob("*.json")), ( + f"Unsupported packaged links asset present under {links_dir}" + ) + + for dash_id in DASHBOARDS: + path = DASHBOARD_DIR / f"{dash_id}.json" + if not path.is_file(): + raise SystemExit(f"Missing {path}") + verify(path) + print(f"OK source {path.name}") + + config_path = DASHBOARD_DIR / f"{CONFIG_ID}.json" + verify_config(config_path) + print(f"OK source {config_path.name}") + + +def zip_dashboard_member(zf: zipfile.ZipFile, dash_id: str) -> str: + suffix = f"kibana/dashboard/{dash_id}.json" + matches = [name for name in zf.namelist() if name.endswith(suffix)] + if len(matches) != 1: + raise SystemExit(f"{zf.filename}: expected one {suffix}, found {matches}") + return matches[0] + + +def verify_zip(zip_path: Path) -> None: + with zipfile.ZipFile(zip_path) as zf: + for dash_id in DASHBOARDS: + member = zip_dashboard_member(zf, dash_id) + raw = zf.read(member) + obj = json.loads(raw) + verify(Path(dash_id + ".json"), obj) + text = raw.decode() + assert SENTINEL in text, f"{zip_path}:{member} missing sentinel" + assert '"selected_options": []' not in text, f"{zip_path}:{member} empty selected_options" + print(f"OK zip {member} ({text.count(SENTINEL)} sentinel refs)") + + config_member = zip_dashboard_member(zf, CONFIG_ID) + verify_config_obj(json.loads(zf.read(config_member)), f"{zip_path}:{config_member}") + print(f"OK zip {config_member}") + + +def main() -> int: + parser = argparse.ArgumentParser() + parser.add_argument("--zip", type=Path, help="Also verify built chargeback zip") + args = parser.parse_args() + + verify_source_tree() + if args.zip: + if not args.zip.is_file(): + print(f"Zip not found: {args.zip}", file=sys.stderr) + return 1 + verify_zip(args.zip) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main())