diff --git a/docs/design/2026-07-01-read-billing-demo-ru-model.md b/docs/design/2026-07-01-read-billing-demo-ru-model.md new file mode 100644 index 0000000000000..87f5474687e93 --- /dev/null +++ b/docs/design/2026-07-01-read-billing-demo-ru-model.md @@ -0,0 +1,711 @@ +# Read Billing Demo RU 模型与校准统计面设计 + +- Author(s): TBD +- Discussion PR: TBD +- Tracking Issue: TBD + +## 目录 + +* [简介](#简介) +* [背景和目标](#背景和目标) +* [当前代码边界](#当前代码边界) +* [模型和支持范围](#模型和支持范围) +* [推荐统计面](#推荐统计面) +* [采集和生命周期](#采集和生命周期) +* [查询和权重校准方法](#查询和权重校准方法) +* [初始权重](#初始权重) +* [性能和容量控制](#性能和容量控制) +* [兼容性和迁移计划](#兼容性和迁移计划) +* [测试设计](#测试设计) +* [备选方案](#备选方案) +* [风险和未决问题](#风险和未决问题) + +## 简介 + +本文描述 read billing demo 的 SELECT 读请求 RU 模型，以及用于 workload 权重校准的 TiDB-native 统计面。 + +模型把读请求执行过程拆成有限数量的 operator class。每个 operator class 都按版本管理权重，权重作用在无系数的 base units 上： + +```text +preview_ru = + fixed_events * fixed_weight + + input_rows * row_weight + + input_bytes * byte_weight +``` + +第一版是校准 demo，不是最终生产 billing 语义。它和现有 RU v2 resource-control 上报保持隔离，并分成两个输出面： + +1. `EXPLAIN ANALYZE FORMAT='RU'`：单语句诊断输出，显示 base units、weight 和 preview RU。 +2. statement-summary-associated statistics tables：workload 级无系数统计面，用于离线重新套任意 weights，不依赖 Prometheus scrape，也不打印 per-SQL log。 + +Prometheus `tidb_read_billing_demo_*` metrics 可以继续作为 bounded observability 辅助，但不能作为 workload testers 的必需接口；校准契约以 SQL 查询面为准。 + +## 背景和目标 + +现有 RU v2 主要是 statement-level billing 逻辑，不适合回答两个问题： + +1. 一条 SELECT 语句里的 TiDB / TiKV 哪些 executor 消耗了 RU； +2. 在 workload 下，如何收集无系数、可重放的原始输入，用真实压测数据回归出更合理的 weights。 + +read billing demo 的目标是： + +1. 让单条语句可以通过 `EXPLAIN ANALYZE FORMAT='RU'` 看到 operator-level 的 base units、weight 和 preview RU； +2. 让 workload 可以通过 SQL 查询收集 `site/op_class/unit` 维度的 `fixed_events`、`input_rows`、`input_bytes` 等基础量； +3. 让 testers 可以在 workload 结束后直接替换 weights 并离线重算 RU，不必重跑 workload； +4. 让 unsupported、unknown input、statement error、aggregation overflow 等失败或降级状态有可查询位置； +5. 在 weights 还没有校准完成前，避免把 demo 结果混入现有 RU v2 billing 或 resource-control consumption 上报。 + +非目标： + +- 不把 per-SQL log 作为校准接口； +- 不要求 downstream testers 搭 Prometheus 或依赖 metrics retention； +- 不在第一版覆盖 TiFlash、MPP、IndexMerge 或复杂 storage path； +- 不把当前 `v1` weights 解释成生产校准值。 + +## 当前代码边界 + +当前实现已经有以下锚点： + +- `pkg/sessionctx/vardef/tidb_vars.go`、`pkg/sessionctx/variable/session.go`、`pkg/sessionctx/variable/sysvar.go` 定义 `tidb_enable_read_billing_demo`，默认 OFF。 +- `pkg/executor/adapter.go::FinishExecuteStmt` 在 statement 完成时调用 `plannercore.RecordReadBillingDemoForStatement`，随后 `SummaryStmt` 把返回的 read-billing summary 放进 `stmtsummary.StmtExecInfo`。 +- `pkg/session/session.go::recordReadBillingDemoEarlyError` 覆盖 compile / resolve / pre-completion error 等没有正常走到 `FinishExecuteStmt` 的错误出口。 +- `pkg/planner/core/explain_ru.go::buildReadBillingDemoResult` 生成 frozen result，包含 statement status、operator status 和 operator base units。 +- `pkg/planner/core/explain_ru.go::recordReadBillingDemoResult` 当前会写 bounded Prometheus metrics。 +- `pkg/planner/core/explain_ru.go::summarizeReadBillingDemoBaseUnits` 当前只把成功 statement 的 base units 汇总成三个 scalar totals。 +- `pkg/util/stmtsummary/statement_summary.go`、`pkg/util/stmtsummary/reader.go`、`pkg/infoschema/tables.go` 当前在 statement summary v1 路径暴露三列： + - `SUM_READ_BILLING_DEMO_FIXED_EVENTS` + - `SUM_READ_BILLING_DEMO_INPUT_ROWS` + - `SUM_READ_BILLING_DEMO_INPUT_BYTES` +- `pkg/util/stmtsummary/v2/record.go`、`pkg/util/stmtsummary/v2/column.go` 当前在 persistent statement summary v2 路径保持同样三列。 + +三列 totals 可以作为 convenience summary，但它们丢失了 `site/op_class/unit/input_source/input_side` 维度，无法支持 per-opclass weight calibration。因此本设计推荐新增维度化 SQL 查询面，并从同一个 statement summary 生命周期里聚合。 + +## 模型和支持范围 + +### 支持范围和安全 gate + +demo 只接受 side-effect-free SELECT： + +- 普通 `SELECT`； +- set operation，前提是所有叶子都是 side-effect-free SELECT； +- prepared statement / `EXECUTE`，前提是 resolved prepared target 是可支持的 side-effect-free SELECT。 + +以下语句或路径会被拒绝、跳过或 fail closed： + +- 非 SELECT； +- `SELECT ... INTO`； +- locking SELECT； +- 带副作用的函数，例如 lock 函数、sequence mutation、`LAST_INSERT_ID(expr)` 和 `SLEEP`； +- internal / restricted SQL； +- TiFlash、MPP exchange、IndexMerge 和未支持的 operator。 + +当出现 unsupported、unknown input 或 statement error 时，demo 必须记录 status/reason，但不能上报 partial base units 或 partial preview RU。 + +### Base-unit result + +每个 supported 且实际执行的 billable operator 至少产生一个： + +```text +fixed_events = 1 +``` + +大多数 operator 还会产生： + +- `input_rows`：该 operator 消耗的行数或 key 数； +- `input_bytes`：该 operator 消耗的 byte-shaped 输入。 + +每个 operator 结果都有以下身份字段： + +- `site`：`tidb`、`tikv`，statement-level status 用 `statement`； +- `op_class`：用于解析权重的有限 class； +- `operator_kind`：真实 plan/operator 名称，只用于观测和调参，不参与 weight lookup； +- `model_version`：当前统计模型版本，当前是 `v1`； +- `weight_version`：当前 weight table 版本，当前是 `v1`。 + +权重按下面的 key 解析： + +```text +site + op_class + weight_version +``` + +因此 TiDB 和 TiKV 即使共享类似的 opclass 名称，例如 `filter_eval`，也可以使用不同权重。 + +### TiKV Cop Executor 模型 + +TiKV 侧不是只按 scan 计费，而是把 cop executor 也纳入模型。第一版包括： + +| TiKV opclass | 覆盖的 operator 形态 | base-unit 输入 | +|---|---|---| +| `kv_range_scan` | table/index range scan | `input_rows = ScanDetail.TotalKeys`；`input_bytes = ScanDetail.ProcessedKeysSize` 优先，缺失时回退 `TotalKeys * row_width` | +| `kv_point_lookup` | point get / batch point get | point lookup work；当前 variable input 基于 runtime rows，fixed event 覆盖 operator setup | +| `filter_eval` | selection | 直接 cop child runtime rows 和 modeled row width | +| `projection_eval` | projection | 直接 cop child runtime rows 和 modeled row width | +| `row_limit` | limit | 直接 cop child runtime rows 和 modeled row width | +| `bounded_topn` | TopN | 直接 cop child runtime rows 和 modeled row width | +| `agg_hash` | hash aggregation | aggregation input rows 和 modeled input bytes | +| `agg_stream` | stream aggregation | aggregation input rows 和 modeled input bytes | + +range scan 的 byte 输入优先使用 `ProcessedKeysSize`，因为 scan 成本不仅和输出行数有关，还和 key/value bytes、MVCC 数据移动、row decode 有关。如果 runtime 只有 `TotalKeys` 而没有 processed size，则回退为： + +```text +input_bytes = TotalKeys * row_width +``` + +如果是 mock-store 或其他没有 scan detail 的路径，则回退为： + +```text +input_rows = actRows +input_bytes = actRows * row_width +input_source = runtime_act_rows +``` + +这个 fallback 会通过 `input_source` 明确标出，避免把 runtime rows 伪装成 scan detail。 + +### TiDB Root Executor 模型 + +TiDB 侧覆盖 root executor CPU、row movement、reader materialization、join 和 local overlay： + +| TiDB opclass | 覆盖的 operator 形态 | base-unit 输入 | +|---|---|---| +| `filter_eval` | root selection | local child rows 和 modeled input bytes | +| `projection_eval` | root projection | local child rows 和 modeled input bytes | +| `row_limit` | root limit | local child rows 和 modeled input bytes | +| `bounded_topn` | root TopN | local child rows 和 modeled input bytes | +| `full_ordering` | sort | local child rows 和 modeled input bytes | +| `window_eval` | window executor | local child rows 和 modeled input bytes | +| `agg_hash` | hash aggregation | aggregation input rows 和 modeled input bytes | +| `agg_stream` | stream aggregation | aggregation input rows 和 modeled input bytes | +| `join_hash` | hash join | build/probe 两侧 input rows 和 bytes | +| `join_merge` | merge join | left/right 两侧 input rows 和 bytes | +| `join_lookup` | index lookup join family | left/right 两侧 input rows 和 bytes | +| `reader_receive` | table/index reader receive | runtime rows 和 modeled received bytes | +| `lookup_reader` | index lookup reader orchestration | runtime rows 和 modeled bytes | +| `overlay_reader` | union scan / local overlay | runtime rows 和 modeled bytes | +| `metadata_reader` | memory / cluster metadata reads | runtime rows 和 modeled bytes | + +Join 的 side 会通过 `input_side` 保留下来： + +- hash join：`build`、`probe`； +- merge join / lookup join：`left`、`right`； +- 其他 operator：`all`。 + +## 推荐统计面 + +推荐新增 statement-summary-associated virtual tables，按 statement summary 的 digest/window/history 生命周期聚合 read billing demo 明细。表名前缀和现有 statement summary 保持一致： + +- `INFORMATION_SCHEMA.STATEMENTS_SUMMARY_READ_BILLING_DEMO_BASE_UNITS` +- `INFORMATION_SCHEMA.STATEMENTS_SUMMARY_HISTORY_READ_BILLING_DEMO_BASE_UNITS` +- `INFORMATION_SCHEMA.CLUSTER_STATEMENTS_SUMMARY_READ_BILLING_DEMO_BASE_UNITS` +- `INFORMATION_SCHEMA.CLUSTER_STATEMENTS_SUMMARY_HISTORY_READ_BILLING_DEMO_BASE_UNITS` +- `INFORMATION_SCHEMA.STATEMENTS_SUMMARY_READ_BILLING_DEMO_STATUS` +- `INFORMATION_SCHEMA.STATEMENTS_SUMMARY_HISTORY_READ_BILLING_DEMO_STATUS` +- `INFORMATION_SCHEMA.CLUSTER_STATEMENTS_SUMMARY_READ_BILLING_DEMO_STATUS` +- `INFORMATION_SCHEMA.CLUSTER_STATEMENTS_SUMMARY_HISTORY_READ_BILLING_DEMO_STATUS` + +### Base-unit table schema + +每一行表示一个 statement summary window 内、一个 digest/plan_digest 下的一组 base-unit key 聚合。 + +| Column | Type | 含义 | +|---|---|---| +| `SUMMARY_BEGIN_TIME` | timestamp | statement summary window begin | +| `SUMMARY_END_TIME` | timestamp | statement summary window end | +| `STMT_TYPE` | varchar | statement type | +| `SCHEMA_NAME` | varchar | schema | +| `DIGEST` | varchar | SQL digest | +| `DIGEST_TEXT` | blob | normalized SQL，沿用 statement summary 截断策略 | +| `PLAN_DIGEST` | varchar | plan digest；point get 可由 lazy plan digest 补齐 | +| `RESOURCE_GROUP` | varchar | resource group | +| `MODEL_VERSION` | varchar | read billing model version | +| `WEIGHT_VERSION` | varchar | 生成这些 base units 时匹配的 weight version | +| `SITE` | varchar | `tidb` / `tikv` | +| `OP_CLASS` | varchar | bounded opclass | +| `OPERATOR_KIND` | varchar | bounded plan/operator kind | +| `UNIT` | varchar | `fixed_events` / `input_rows` / `input_bytes` | +| `INPUT_SOURCE` | varchar | `scan_detail` / `runtime_act_rows` | +| `INPUT_SIDE` | varchar | `all` / `build` / `probe` / `left` / `right` | +| `ROW_WIDTH_SOURCE` | varchar | `plan_stats` / `schema_type_width` / `schema_fallback` / `operator_helper` | +| `VALUE` | double unsigned | 该 key 的 base-unit total | +| `SAMPLE_COUNT` | bigint unsigned | 汇入该 key 的 unit samples 数 | +| `ROW_WIDTH_SUM` | double unsigned | row-width sample sum，用于诊断 modeled bytes | +| `AVG_ROW_WIDTH` | double unsigned | `ROW_WIDTH_SUM / SAMPLE_COUNT` | + +`DIGEST_TEXT` 不是 key，只是查询便利字段。校准时推荐按 `DIGEST`、`PLAN_DIGEST`、`SITE`、`OP_CLASS`、`UNIT` 等机器字段聚合。 + +### Status table schema + +每一行表示一个 statement summary window 内、一个 digest/plan_digest 下的一组 statement/operator status 聚合。 + +| Column | Type | 含义 | +|---|---|---| +| `SUMMARY_BEGIN_TIME` | timestamp | statement summary window begin | +| `SUMMARY_END_TIME` | timestamp | statement summary window end | +| `STMT_TYPE` | varchar | statement type | +| `SCHEMA_NAME` | varchar | schema | +| `DIGEST` | varchar | SQL digest | +| `DIGEST_TEXT` | blob | normalized SQL | +| `PLAN_DIGEST` | varchar | plan digest | +| `RESOURCE_GROUP` | varchar | resource group | +| `MODEL_VERSION` | varchar | read billing model version | +| `WEIGHT_VERSION` | varchar | current weight version at accounting time | +| `SITE` | varchar | `statement` / `tidb` / `tikv` | +| `OP_CLASS` | varchar | statement or operator opclass | +| `OPERATOR_KIND` | varchar | statement or plan/operator kind | +| `STATUS` | varchar | `success` / `unsupported` / `unknown_input` / `error` / `ok` | +| `REASON` | varchar | bounded reason，例如 `unsupported_mpp`、`missing_scan_detail`、`statement_error` | +| `COUNT` | bigint unsigned | 该 status key 的出现次数 | + +失败 statement 只写 status table，不写 base-unit table。成功 statement 写一条 statement-level `success` status，并为每个 billable operator 写 operator-level `ok` status。 + +### 为什么不用 statement summary 单表 JSON 列 + +JSON/encoded column 的写入成本可以较低，但对校准不够直接： + +- testers 需要额外 JSON 解析才能 group by `site/op_class/unit`； +- SQL 里直接套 weights、按 digest/window 过滤和 join 会变复杂； +- 历史持久化中的 schema 演进、旧 JSON 兼容和 partial parse failure 更难测试； +- 失败 status 和 base units 混在一个 encoded blob 里，容易再次变成不可直接观测的状态。 + +窄表把 row explosion 放到查询阶段，而不是把复杂性塞进单列解析；同时它复用 statement summary 的 digest/window/LRU/history 控制。 + +## 采集和生命周期 + +### Statement completion path + +后续实现应把 `RecordReadBillingDemoForStatement` 的返回值从三个 totals 扩成结构化 snapshot，例如： + +```go +type ReadBillingDemoStatementStats struct { + ModelVersion string + WeightVersion string + Statuses []ReadBillingDemoStatusSample + BaseUnits []ReadBillingDemoBaseUnitSample + Totals ReadBillingDemoBaseUnitSummary +} +``` + +其中 `Totals` 从 `BaseUnits` 派生，用于继续填充现有三列；`Statuses` 和 `BaseUnits` 则进入新表所需的维度化聚合。 + +采集顺序： + +1. statement 执行结束或早期错误出口触发 read billing demo hook； +2. `buildReadBillingDemoResult` 生成 frozen result； +3. 如果 result status 是 success： + - 为 statement 写 `success` status； + - 为 billable operator 写 operator `ok` status； + - 把 operator units 展平成 base-unit samples； + - 从 base-unit samples 派生三列 totals； +4. 如果 result status 不是 success： + - 为 statement 或 failed operator 写 status/reason； + - 不写任何 base-unit samples； + - 三列 totals 保持 0； +5. 正常执行结束路径由 `ExecStmt.SummaryStmt` 把结构化 snapshot 挂到 `StmtExecInfo`； +6. statement summary v1/v2 通过同一个 `Add`/`Merge` 路径聚合到当前 window 和 history； +7. 早期错误路径不能假设一定有 `ExecStmt.SummaryStmt`。`session.recordReadBillingDemoEarlyError` 需要在调用 `RecordReadBillingDemoForStatement` 后，把 status-only snapshot 通过一个 read-billing 专用 helper 写入同一套 statement-summary-associated status 聚合，例如 `stmtsummaryv2.AddReadBillingDemoStatusOnly(...)`。该 helper 只记录 status table 所需字段，`PLAN_DIGEST` 可为空，不写 base units，不影响普通 statement summary latency/RU 等通用列。 + +### Statement summary v1 + +v1 in-memory path 位于 `pkg/util/stmtsummary/statement_summary.go`： + +- `stmtSummaryStats` 新增 bounded in-memory maps： + - `ReadBillingDemoBaseUnitAggs map[ReadBillingDemoBaseUnitKey]ReadBillingDemoBaseUnitAgg` + - `ReadBillingDemoStatusAggs map[ReadBillingDemoStatusKey]ReadBillingDemoStatusAgg` +- `newStmtSummaryStats`、`stmtSummaryStats.add`、`ReadBillingDemo...Merge` 负责把 `StmtExecInfo` 的 samples 合并进去； +- status-only early error helper 使用同样的 key 构造和 window 选择逻辑，但只 upsert status map；如果 statement digest 不可用，则使用空 digest/空 plan digest 聚合到当前 window，并保留 bounded reason； +- `stmtSummaryReader` 新增面向 base-unit/status 表的 reader，遍历同一个 current/history/cumulative window 后展开 rows； +- `STATEMENTS_SUMMARY_READ_BILLING_DEMO_*` 表沿用 statement summary 的 auth 行为：无 `PROCESS` privilege 时只能看到自己的 statement 聚合。 + +### Statement summary v2 + +v2 persistent path 位于 `pkg/util/stmtsummary/v2`： + +- `StmtRecord` 新增 JSON-stable entry slices，而不是 `map[struct]...` 这类不能稳定 marshal/unmarshal 的字段： + - `ReadBillingDemoBaseUnitAggs []ReadBillingDemoBaseUnitAggEntry` + - `ReadBillingDemoStatusAggs []ReadBillingDemoStatusAggEntry` +- 每个 entry 直接携带 key 字段和 aggregate value；`StmtRecord.Add` / `StmtRecord.Merge` 可以用临时 canonical string key 或小规模线性 upsert 合并后再保持 entry slice 排序； +- `MemReader` 从当前 `StmtRecord` 展开新表 rows； +- `HistoryReader` 读取历史 JSON log 时，如果旧记录没有这些字段，则 entry slices 为 nil，新表返回 0 行，不影响旧 log 解析； +- evicted aggregate record 也合并 read billing entries。对校准而言，evicted row 只作为覆盖率/容量诊断，不应用来拟合精细 weights，因为 digest/plan 维度已经被合并。 + +### Current 和 history 表 + +当前表读取未持久化窗口，history 表读取内存当前窗口加已持久化历史，与现有 `STATEMENTS_SUMMARY` / `STATEMENTS_SUMMARY_HISTORY` 语义保持一致。 + +如果 user 需要完整 workload run，推荐： + +1. 开启 statement summary； +2. 确认 `tidb_enable_read_billing_demo = ON`； +3. 记录 workload 起止时间； +4. 从 `CLUSTER_STATEMENTS_SUMMARY_HISTORY_READ_BILLING_DEMO_*` 用 `SUMMARY_BEGIN_TIME` / `SUMMARY_END_TIME` 过滤； +5. 同时查 status 表确认失败和 overflow 比例。 + +## 查询和权重校准方法 + +### 开启采集 + +```sql +SET GLOBAL tidb_enable_read_billing_demo = ON; +``` + +建议只在校准 workload 的 session 或测试集群中开启。生产默认仍然是 OFF。 + +### 第一步：检查覆盖率和失败状态 + +先看 statement/operator status： + +```sql +SELECT + status, + reason, + SUM(count) AS count +FROM information_schema.cluster_statements_summary_history_read_billing_demo_status +WHERE summary_begin_time >= TIMESTAMP '2026-07-02 10:00:00' + AND summary_end_time <= TIMESTAMP '2026-07-02 11:00:00' + AND model_version = 'v1' +GROUP BY status, reason +ORDER BY count DESC; +``` + +如果 `unsupported`、`unknown_input`、`error` 或 `aggregation_overflow` 占比高，说明 base-unit 数据不能代表整个 workload，需要先收敛覆盖范围或降低维度压力。 + +进一步按 operator 看原因： + +```sql +SELECT + site, + op_class, + operator_kind, + status, + reason, + SUM(count) AS count +FROM information_schema.cluster_statements_summary_history_read_billing_demo_status +WHERE summary_begin_time >= TIMESTAMP '2026-07-02 10:00:00' + AND summary_end_time <= TIMESTAMP '2026-07-02 11:00:00' + AND model_version = 'v1' +GROUP BY site, op_class, operator_kind, status, reason +ORDER BY count DESC; +``` + +### 第二步：收集 base units + +按 `site/op_class/unit/source/side` 聚合 base units： + +```sql +SELECT + site, + op_class, + unit, + input_source, + input_side, + SUM(value) AS value +FROM information_schema.cluster_statements_summary_history_read_billing_demo_base_units +WHERE summary_begin_time >= TIMESTAMP '2026-07-02 10:00:00' + AND summary_end_time <= TIMESTAMP '2026-07-02 11:00:00' + AND model_version = 'v1' +GROUP BY site, op_class, unit, input_source, input_side +ORDER BY site, op_class, unit, input_source, input_side; +``` + +这个查询是校准模型的主要输入，相当于得到无系数特征矩阵： + +```text +X[site, op_class, unit] = 当前时间窗口内观测到的 base-unit total +``` + +如果需要区分 plan shape，可以保留 `DIGEST` 和 `PLAN_DIGEST`： + +```sql +SELECT + digest, + plan_digest, + site, + op_class, + unit, + SUM(value) AS value +FROM information_schema.cluster_statements_summary_history_read_billing_demo_base_units +WHERE summary_begin_time >= TIMESTAMP '2026-07-02 10:00:00' + AND summary_end_time <= TIMESTAMP '2026-07-02 11:00:00' + AND model_version = 'v1' +GROUP BY digest, plan_digest, site, op_class, unit; +``` + +### 第三步：检查 row-width evidence + +row width 是模型 factor，不是 runtime 真实 byte sample。拟合前需要检查它是否稳定： + +```sql +SELECT + site, + op_class, + row_width_source, + SUM(sample_count) AS samples, + SUM(row_width_sum) / NULLIF(SUM(sample_count), 0) AS avg_row_width +FROM information_schema.cluster_statements_summary_history_read_billing_demo_base_units +WHERE summary_begin_time >= TIMESTAMP '2026-07-02 10:00:00' + AND summary_end_time <= TIMESTAMP '2026-07-02 11:00:00' + AND unit = 'input_bytes' + AND model_version = 'v1' +GROUP BY site, op_class, row_width_source +ORDER BY site, op_class, row_width_source; +``` + +如果 `schema_fallback` 占比高，或某些 opclass 的 row width 异常，拟合出来的 byte weight 可能是在补偿 row-width 估计误差，而不是真实 byte cost。 + +### 第四步：用任意 weights 重算 RU + +testers 可以把新 weights 放到 CTE、临时表或外部 notebook。SQL 侧的最小形态如下： + +```sql +WITH observed AS ( + SELECT site, op_class, unit, SUM(value) AS value + FROM information_schema.cluster_statements_summary_history_read_billing_demo_base_units + WHERE summary_begin_time >= TIMESTAMP '2026-07-02 10:00:00' + AND summary_end_time <= TIMESTAMP '2026-07-02 11:00:00' + AND model_version = 'v1' + GROUP BY site, op_class, unit +), +weights(site, op_class, fixed_weight, row_weight, byte_weight) AS ( + SELECT 'tikv', 'kv_range_scan', 0.070, 0.000045, 0.000020 UNION ALL + SELECT 'tidb', 'projection_eval', 0.020, 0.000020, 0.000004 +) +SELECT + SUM(CASE observed.unit + WHEN 'fixed_events' THEN observed.value * weights.fixed_weight + WHEN 'input_rows' THEN observed.value * weights.row_weight + WHEN 'input_bytes' THEN observed.value * weights.byte_weight + ELSE 0 + END) AS recalculated_ru +FROM observed +JOIN weights + ON observed.site = weights.site + AND observed.op_class = weights.op_class; +``` + +因为统计面保留的是 base units，重算时只需要替换 `weights` 数据，不需要重新跑 workload。 + +### 第五步：拟合新 weights + +对每个校准窗口构造： + +```text +y_window = 观测目标成本 +X_window = 按 site/opclass/unit 聚合的 base-unit totals +``` + +`y_window` 可以选择： + +- workload 窗口内 TiDB/TiKV CPU time； +- 现有 RU consumption，用于兼容性对比； +- benchmark 手工标注成本； +- 混合目标，例如 CPU + IO-normalized scan cost。 + +推荐使用非负线性模型： + +```text +minimize || X * w - y ||^2 +subject to w >= 0 +``` + +拟合规则建议： + +1. 按 `site/op_class` 分别拟合 `fixed`、`row`、`byte` 三类系数。 +2. 保持权重非负。 +3. 使用 hold-out workload window 验证预测误差。 +4. 默认保留符合源码成本形态的单调关系，除非数据强烈反证，例如： + - `agg_hash.row >= agg_stream.row`； + - `bounded_topn.row >= row_limit.row`； + - `kv_range_scan.byte >= filter_eval.byte`； + - `join_hash.row >= join_merge.row`。 +5. 拟合后的新权重应该发布为新的 `weight_version`，不要静默改变 `v1` 的含义。 + +## 初始权重 + +当前 `v1` weights 是 demo calibration seed，不是 production-calibrated billing constants。它们来自源码成本形态和相对排序： + +- scan 同时受 row/key 数和 bytes 影响，比纯 expression eval 更重； +- hash aggregation 和 hash join 需要维护 hash map / hash table / aggregate state，row weight 更高； +- stream aggregation 和 merge join 依赖有序输入，通常低于 hash 版本； +- limit 是最轻的 row-shaped operator； +- TopN 需要 order expression 和 heap maintenance，比 limit 重； +- TiDB reader class 对 bytes 更敏感，因为它负责接收、materialize chunk 和协调 lookup； +- TiDB 与 TiKV 即使 opclass 名称相同，也运行不同代码路径，因此权重分开设置。 + +当前 `v1` 权重如下： + +| site | opclass | fixed_weight | row_weight | byte_weight | 初始设置原因 | +|---|---|---:|---:|---:|---| +| `tikv` | `kv_range_scan` | 0.070 | 0.000045 | 0.000020 | range iterator、MVCC/key-value movement、row decode 和 scan limiter 使 scan 同时对 row/key 和 bytes 敏感。 | +| `tikv` | `kv_point_lookup` | 0.045 | 0.000030 | 0.000012 | point get 有 per-key dispatch 和 decode，但没有 range iterator loop。 | +| `tikv` | `filter_eval` | 0.020 | 0.000040 | 0.000006 | RPN predicate eval 主要按 logical rows 消耗 CPU，byte movement 权重较低。 | +| `tikv` | `projection_eval` | 0.020 | 0.000030 | 0.000006 | projection 会执行表达式或 materialize column reference，row 成本低于 filter。 | +| `tikv` | `row_limit` | 0.010 | 0.000008 | 0.000002 | 主要是 pass-through counter / gate，是最轻的 executor。 | +| `tikv` | `bounded_topn` | 0.060 | 0.000075 | 0.000012 | order expression eval 加 bounded heap maintenance。 | +| `tikv` | `agg_hash` | 0.080 | 0.000100 | 0.000014 | group expression eval、hash group lookup/insert、aggregate state update。 | +| `tikv` | `agg_stream` | 0.060 | 0.000065 | 0.000010 | ordered group compare 加 state update，不构建 hash table。 | +| `tidb` | `filter_eval` | 0.020 | 0.000030 | 0.000005 | TiDB chunk 上的 expression eval，低于 scan 和 hash 类 operator。 | +| `tidb` | `projection_eval` | 0.020 | 0.000020 | 0.000004 | projection / materialization，per-row 成本较低。 | +| `tidb` | `row_limit` | 0.010 | 0.000006 | 0.000001 | 轻量 pass-through control。 | +| `tidb` | `bounded_topn` | 0.060 | 0.000060 | 0.000010 | heap / order expression work，比 full sort 范围小。 | +| `tidb` | `full_ordering` | 0.080 | 0.000070 | 0.000012 | full sort 需要处理完整输入。 | +| `tidb` | `window_eval` | 0.070 | 0.000070 | 0.000010 | partition / order / frame state maintenance。 | +| `tidb` | `agg_hash` | 0.070 | 0.000085 | 0.000012 | hash grouping 和 aggregate state update。 | +| `tidb` | `agg_stream` | 0.050 | 0.000055 | 0.000008 | ordered streaming aggregate，低于 hash agg。 | +| `tidb` | `join_hash` | 0.110 | 0.000115 | 0.000020 | build/probe hash table work，是 TiDB 侧最高 row cost 之一。 | +| `tidb` | `join_merge` | 0.090 | 0.000075 | 0.000012 | ordered comparison / merge，低于 hash join。 | +| `tidb` | `join_lookup` | 0.120 | 0.000120 | 0.000020 | lookup task orchestration 加 join-side work，固定成本和 row 成本都较高。 | +| `tidb` | `reader_receive` | 0.040 | 0.000025 | 0.000014 | network/result receive 和 chunk materialization，对 bytes 敏感。 | +| `tidb` | `lookup_reader` | 0.070 | 0.000045 | 0.000016 | index/table two-phase lookup reader coordination。 | +| `tidb` | `overlay_reader` | 0.050 | 0.000035 | 0.000012 | UnionScan/local overlay merge。 | +| `tidb` | `metadata_reader` | 0.020 | 0.000008 | 0.000002 | metadata / memory-table read，故意设得较轻。 | + +## 性能和容量控制 + +### 低开销原则 + +新统计面必须满足： + +- `tidb_enable_read_billing_demo` 默认 OFF，关闭时不构造 read billing result，也不写 statement summary maps； +- 开启后仍只在 statement end 聚合一次，不打印 per-SQL log； +- base-unit/status key 由 bounded enum-like 字段组成，不包含 SQL text、table name、index name、region、store address 或 plan id； +- `DIGEST_TEXT` 和 `PLAN_DIGEST` 复用 statement summary 已有字段，不进入 read-billing map key； +- 读表时才把 map 展开成多行，写入时保持 map 聚合，不为每次执行追加 unbounded slice； +- cluster 表沿用现有 cluster memtable fan-out，不引入新的后台 scrape。 + +### 单 digest/window map 上限 + +statement summary 已有 `max-stmt-count`、`refresh-interval`、`history-size` 和 LRU eviction，但这些只限制 digest/window 数量。read billing 还需要限制单 digest/window 内的维度 key 数。 + +建议第一版使用保守上限： + +```text +max_base_unit_keys_per_record = 256 +max_status_keys_per_record = 128 +``` + +当超过上限： + +- status table 增加 `status='unknown_input'`、`reason='aggregation_overflow'` 的 reserved row； +- 超出上限的新 base-unit key 不再写入 base-unit table，避免把不完整特征矩阵伪装成完整数据； +- 现有已聚合 key 继续累加； +- 三列 totals 仍只从已接收 base-unit samples 派生，并且用户必须通过 status 表判断该窗口是否可用于拟合。 + +overflow row 必须永远可见，不能被 status map 自身的上限挤掉。实现时预留两个不计入 `max_status_keys_per_record` 的 fixed buckets： + +```text +statement/statement/statement unknown_input aggregation_overflow +statement/statement/statement unknown_input status_aggregation_overflow +``` + +如果 base-unit map 超限，写 `aggregation_overflow` bucket。如果 status map 对普通 status key 超限，写 `status_aggregation_overflow` bucket。reserved buckets 只累加 count，不触发新的 overflow，也不写 base units。 + +这个策略牺牲超高维异常 workload 的完整性，但保证整体性能和内存不会被单个 digest 放大。 + +### 估算内存上界 + +粗略上界： + +```text +max_records = stmt-summary.max-stmt-count * (history-size 或 current window) +per_record_read_billing_keys <= 256 + 128 +``` + +实际 key 数通常远低于上限，因为第一版 opclass、unit、input_source 和 input_side 都是有限集合。典型 SELECT digest/window 只会产生数十个 key。 + +## 兼容性和迁移计划 + +1. 保留 `SUM_READ_BILLING_DEMO_FIXED_EVENTS`、`SUM_READ_BILLING_DEMO_INPUT_ROWS`、`SUM_READ_BILLING_DEMO_INPUT_BYTES` 三列作为 convenience totals。 +2. 文档和后续测试明确：三列不能用于 per-opclass calibration，因为它们丢失维度。 +3. 新 base-unit/status 表成为 workload calibration 的主接口。 +4. Prometheus `tidb_read_billing_demo_*` metrics 可继续保留为 optional observability，但不作为 testers 必需依赖。 +5. v2 persisted statement summary 的 JSON 结构新增 read-billing maps；旧日志没有这些字段时读出来为空，不需要 migration。 +6. 如果后续模型稳定并准备生产化，可以再决定是否移除 demo 三列或把 demo 表改名为非 demo 表；第一版不做破坏性删除。 + +## 测试设计 + +后续实现 loop 至少需要补以下测试。 + +### Planner / calculator tests + +- `RecordReadBillingDemoForStatement` 返回结构化 status/base-unit snapshot； +- success path 只为 billable operators 产生 base units； +- unsupported、unknown-input、error path 不产生 base units，但产生 status/reason； +- normal finish path 通过 `StmtExecInfo` 写 status/base units，early error path 通过 status-only helper 写 SQL-visible status； +- `aggregation_overflow` 和 `status_aggregation_overflow` reserved buckets 可见且不会写 partial unknown key； +- current three-column totals 从 base-unit samples 派生，与现有测试保持兼容。 + +### Statement summary v1 tests + +- `stmtSummaryStats.add` 能按 `site/op_class/unit/input_source/input_side/row_width_source` 聚合 value； +- status-only early error helper 能写入 current window 的 status table，且不写 base-unit rows； +- `Merge` / evicted aggregate 能合并 read billing maps； +- `STATEMENTS_SUMMARY_READ_BILLING_DEMO_BASE_UNITS` current/history rows 与 digest/window 对齐； +- `STATEMENTS_SUMMARY_READ_BILLING_DEMO_STATUS` 能看到 failed billing； +- base-unit/status map 超限时 reserved overflow buckets 不受 cap 影响； +- auth 行为与 statement summary 保持一致。 + +### Statement summary v2 tests + +- `StmtRecord.Add` / `Merge` 聚合 JSON-stable read billing entry slices； +- current reader 展开 base-unit/status rows； +- history reader 能读取新 JSON 字段； +- old history JSON 没有 read-billing 字段时返回 0 行且不报错； +- entry slice marshal/unmarshal 后 key 字段和值不丢失，排序保持 deterministic； +- evicted aggregate 包含 read billing maps，但测试中明确它不适合精细拟合。 + +### Information schema / cluster table tests + +- 新表注册、列顺序、cluster `INSTANCE` 列； +- digest/time predicate extractor 复用 statement summary extractor； +- `CLUSTER_*` 表能查询多个 TiDB instance 的 rows； +- 默认 OFF 时新表无 read-billing rows； +- 开启 demo 后可以用 SQL 直接按 `site/op_class/unit` 重算 RU。 + +### 性能验证 + +- targeted benchmark 比较 demo OFF、demo ON + scalar totals、demo ON + dimension maps； +- 验证正常 workload 下 statement end 额外开销不显著； +- 验证高维异常 workload 触发 overflow 后内存和写入开销保持 bounded。 + +## 备选方案 + +### 继续只用 Prometheus metrics + +不采用。metrics label shape 可以保存 `site/op_class/unit`，但 downstream testers 明确不希望依赖 Prometheus；metrics retention、scrape interval 和 workload 窗口对齐也会增加校准流程复杂度。 + +### 每条 SQL 打日志 + +不采用。per-SQL log 会显著影响 QPS，也会把校准流程变成日志采集和解析问题。 + +### 只扩展现有 statement summary 三列 + +不采用。三列 totals 丢失 `site/op_class/unit`，只能重算一个全局粗粒度模型，不能直接优化不同 opclass 的 weights。 + +### 在 statement summary 单表里放 JSON/encoded column + +暂不采用。它减少表数量，但把校准所需的 group-by 和 join 复杂度推给用户，也让 failed billing status 不够直接。可以作为后续低表面面积方案重新评估。 + +### 每个 operator kind 一个 weight + +不采用。这样会显著提高模型和统计面复杂度，也会让校准更脆弱。当前保留 `operator_kind` 用于观测，但 weight 绑定在有限 opclass 上。 + +### workload 统计面直接上报 preview RU + +不采用。preview RU 是加权结果，会遮住校准需要的 base-unit 原始证据。workload 表上报 base units；单语句调试通过 `EXPLAIN ANALYZE FORMAT='RU'` 看 preview RU。 + +## 风险和未决问题 + +### 风险 + +- 新表会增加 statement summary 内存占用；必须用单 digest/window key 上限和 overflow status 控制。 +- v2 persistent history 会扩大 statement summary JSON record；需要 benchmark 和 history-reader 兼容测试。 +- row-width factor 仍然是 modeled value，如果大量来自 `schema_fallback`，byte weight 可能补偿 row-width 误差。 +- evicted aggregate 会丢 digest/plan 细节，不适合作为精细校准输入；用户需要查询 status 和 evicted 表判断窗口质量。 +- 当前 point lookup miss 的 variable row cost 证据仍有限；没有 requested-key 证据时主要由 fixed event 覆盖。 + +### 未决问题 + +- 第一轮生产化拟合应该选什么目标信号：CPU、现有 RU、latency-normalized cost，还是混合目标？ +- point lookup 是否需要接入 requested key count，使 key miss 也能产生 variable row cost？ +- scan bytes 后续是否应该拆成 total key size、processed key size、value size 等多个 bounded units？ +- `aggregation_overflow` 的默认上限是否需要暴露为配置，还是作为 demo 常量即可？ +- TiFlash、MPP、IndexMerge 何时从 fail-closed 进入 supported opclasses？ diff --git a/pkg/executor/adapter.go b/pkg/executor/adapter.go index b3a826f033e9a..79c20aff81045 100644 --- a/pkg/executor/adapter.go +++ b/pkg/executor/adapter.go @@ -366,11 +366,12 @@ type ExecStmt struct { Ctx sessionctx.Context // LowerPriority represents whether to lower the execution priority of a query. - LowerPriority bool - isPreparedStmt bool - isSelectForUpdate bool - retryCount uint - retryStartTime time.Time + LowerPriority bool + isPreparedStmt bool + isSelectForUpdate bool + resolvedPreparedStmt ast.StmtNode + retryCount uint + retryStartTime time.Time // Phase durations are splited into two parts: 1. trying to lock keys (but // failed); 2. the final iteration of the retry loop. Here we use @@ -770,11 +771,19 @@ func (a *ExecStmt) inheritContextFromExecuteStmt() { a.Ctx.SetValue(sessionctx.QueryString, executePlan.Stmt.Text()) a.OutputNames = executePlan.OutputNames() a.isPreparedStmt = true + a.resolvedPreparedStmt = executePlan.Stmt a.Plan = executePlan.Plan a.Ctx.GetSessionVars().StmtCtx.SetPlan(executePlan.Plan) } } +func (a *ExecStmt) readBillingDemoStmtNode() ast.StmtNode { + if a.resolvedPreparedStmt != nil { + return a.resolvedPreparedStmt + } + return a.StmtNode +} + func (a *ExecStmt) getSQLForProcessInfo() string { sql := a.Text() if simple, ok := a.Plan.(*plannercore.Simple); ok && simple.Statement != nil { @@ -1689,10 +1698,14 @@ func (a *ExecStmt) FinishExecuteStmt(txnTS uint64, err error, hasMoreResults boo } a.finalizeStatementRUV2Metrics() + // Lazy SELECT metrics are emitted when the record set is closed. If a + // client abandons a result without closing/draining it, this first demo can + // miss that statement instead of guessing an incomplete status. + readBillingDemoStats := plannercore.RecordReadBillingDemoForStatement(a.Ctx, a.Plan, a.readBillingDemoStmtNode(), err) a.updateNetworkTrafficStatsAndMetrics() // `LowSlowQuery` and `SummaryStmt` must be called before recording `PrevStmt`. a.LogSlowQuery(txnTS, succ, hasMoreResults) - a.SummaryStmt(succ) + a.SummaryStmt(succ, readBillingDemoStats) a.observeStmtFinishedForTopProfiling() a.UpdatePlanCacheRuntimeInfo() if sessVars.StmtCtx.IsTiFlash.Load() { @@ -2179,7 +2192,7 @@ func (digest planDigestAlias) planDigestDumpTriggerCheck(config *traceevent.Dump } // SummaryStmt collects statements for information_schema.statements_summary -func (a *ExecStmt) SummaryStmt(succ bool) { +func (a *ExecStmt) SummaryStmt(succ bool, readBillingDemoStats stmtsummary.ReadBillingDemoStatementStats) { sessVars := a.Ctx.GetSessionVars() var userString string if sessVars.User != nil { @@ -2283,6 +2296,8 @@ func (a *ExecStmt) SummaryStmt(succ bool) { stmtExecInfo.KeyspaceID = keyspaceID stmtExecInfo.RUDetail = ruDetail stmtExecInfo.TotalRUV2 = calculateStatementTotalRUV2(sessVars.RUV2Metrics, sessVars.RUV2Weights(), ruDetail) + stmtExecInfo.ReadBillingDemoStats = readBillingDemoStats + stmtExecInfo.ReadBillingDemoBaseUnits = readBillingDemoStats.Totals stmtExecInfo.ResourceGroupName = sessVars.StmtCtx.ResourceGroupName stmtExecInfo.CPUUsages = sessVars.SQLCPUUsages.GetCPUUsages() stmtExecInfo.PlanCacheUnqualified = sessVars.StmtCtx.PlanCacheUnqualified() diff --git a/pkg/executor/builder.go b/pkg/executor/builder.go index 7644929184311..8ffa1e7c5f473 100644 --- a/pkg/executor/builder.go +++ b/pkg/executor/builder.go @@ -2616,10 +2616,18 @@ func (b *executorBuilder) buildMemTable(v *physicalop.PhysicalMemTable) exec.Exe case strings.ToLower(infoschema.TableStatementsSummary), strings.ToLower(infoschema.TableStatementsSummaryHistory), strings.ToLower(infoschema.TableStatementsSummaryEvicted), + strings.ToLower(infoschema.TableStatementsSummaryReadBillingDemoBaseUnits), + strings.ToLower(infoschema.TableStatementsSummaryHistoryReadBillingDemoBaseUnits), + strings.ToLower(infoschema.TableStatementsSummaryReadBillingDemoStatus), + strings.ToLower(infoschema.TableStatementsSummaryHistoryReadBillingDemoStatus), strings.ToLower(infoschema.TableTiDBStatementsStats), strings.ToLower(infoschema.ClusterTableStatementsSummary), strings.ToLower(infoschema.ClusterTableStatementsSummaryHistory), strings.ToLower(infoschema.ClusterTableStatementsSummaryEvicted), + strings.ToLower(infoschema.ClusterTableStatementsSummaryReadBillingDemoBaseUnits), + strings.ToLower(infoschema.ClusterTableStatementsSummaryHistoryReadBillingDemoBaseUnits), + strings.ToLower(infoschema.ClusterTableStatementsSummaryReadBillingDemoStatus), + strings.ToLower(infoschema.ClusterTableStatementsSummaryHistoryReadBillingDemoStatus), strings.ToLower(infoschema.ClusterTableTiDBStatementsStats): var extractor *plannercore.StatementsSummaryExtractor if v.Extractor != nil { diff --git a/pkg/executor/compiler.go b/pkg/executor/compiler.go index b686423358f67..c478dacd5331a 100644 --- a/pkg/executor/compiler.go +++ b/pkg/executor/compiler.go @@ -153,6 +153,9 @@ func (c *Compiler) Compile(ctx context.Context, stmtNode ast.StmtNode) (_ *ExecS OutputNames: names, Ti: &TelemetryInfo{}, } + if preparedObj != nil && preparedObj.PreparedAst != nil { + stmt.resolvedPreparedStmt = preparedObj.PreparedAst.Stmt + } // Use cached plan if possible. if preparedObj != nil && plannercore.IsSafeToReusePointGetExecutor(c.Ctx, is, preparedObj) { if exec, isExec := finalPlan.(*plannercore.Execute); isExec { diff --git a/pkg/executor/explain_test.go b/pkg/executor/explain_test.go index 8d7e04e436d6b..f5e848c09af5e 100644 --- a/pkg/executor/explain_test.go +++ b/pkg/executor/explain_test.go @@ -16,8 +16,11 @@ package executor_test import ( "bytes" + "context" "encoding/json" "fmt" + "os" + "path/filepath" "regexp" "strconv" "strings" @@ -25,9 +28,15 @@ import ( "time" "github.com/pingcap/tidb/pkg/config" + "github.com/pingcap/tidb/pkg/kv" + "github.com/pingcap/tidb/pkg/metrics" + "github.com/pingcap/tidb/pkg/parser/auth" plannercore "github.com/pingcap/tidb/pkg/planner/core" "github.com/pingcap/tidb/pkg/testkit" "github.com/pingcap/tidb/pkg/types" + "github.com/pingcap/tidb/pkg/util/sqlexec" + "github.com/prometheus/client_golang/prometheus" + dto "github.com/prometheus/client_model/go" "github.com/stretchr/testify/require" ) @@ -496,6 +505,7 @@ func TestExplainFormatInCtx(t *testing.T) { types.ExplainFormatTiDBJSON, types.ExplainFormatCostTrace, types.ExplainFormatPlanCache, + types.ExplainFormatRU, } tk.MustExec("select * from t") @@ -515,6 +525,373 @@ func TestExplainFormatInCtx(t *testing.T) { } } +func TestExplainAnalyzeFormatRUOutput(t *testing.T) { + store := testkit.CreateMockStore(t) + tk := testkit.NewTestKit(t, store) + tk.MustExec("use test") + + rows := tk.MustQuery("explain analyze format='ru' select 1").Rows() + require.NotEmpty(t, rows) + require.Equal(t, "summary", rows[0][0]) + require.Equal(t, "total_preview_ru", rows[0][2]) + require.Equal(t, "summary_total", rows[0][15]) + requireExplainRUPlanRow(t, rows) + + tk.MustExec("drop table if exists explain_ru_t") + tk.MustExec("create table explain_ru_t(a int primary key, b varchar(20))") + tk.MustExec("insert into explain_ru_t values (1, 'x'), (2, 'yy')") + rows = tk.MustQuery("explain analyze format='ru' select * from explain_ru_t where a > 0").Rows() + requireExplainRUPlanRow(t, rows) + requireExplainRUOperatorClass(t, rows, "tikv/kv_range_scan") + + rows = tk.MustQuery("explain analyze format='ru' select * from explain_ru_t where a = 1").Rows() + requireExplainRUWeightedOperatorClass(t, rows, "tikv/kv_point_lookup") +} + +func TestExplainAnalyzeFormatRUTiKVCopOperatorClasses(t *testing.T) { + store := testkit.CreateMockStore(t) + tk := testkit.NewTestKit(t, store) + tk.MustExec("use test") + tk.MustExec("drop table if exists explain_ru_cop") + tk.MustExec("create table explain_ru_cop(a int primary key, b int, c varchar(20), key idx_b(b))") + tk.MustExec("insert into explain_ru_cop values (1, 10, 'a'), (2, 20, 'bb'), (3, 20, 'ccc'), (4, 30, 'dddd')") + + cases := []struct { + sql string + opClasses []string + }{ + { + sql: "explain analyze format='ru' select * from explain_ru_cop ignore index(idx_b) where b > 10", + opClasses: []string{ + "tikv/filter_eval", + "tikv/kv_range_scan", + }, + }, + { + sql: "explain analyze format='ru' select a from explain_ru_cop where b > 10", + opClasses: []string{ + "tikv/projection_eval", + "tikv/kv_range_scan", + }, + }, + { + sql: "explain analyze format='ru' select * from explain_ru_cop limit 2", + opClasses: []string{ + "tikv/row_limit", + "tikv/kv_range_scan", + }, + }, + { + sql: "explain analyze format='ru' select * from explain_ru_cop ignore index(idx_b) order by c limit 2", + opClasses: []string{ + "tikv/bounded_topn", + "tikv/kv_range_scan", + }, + }, + { + sql: "explain analyze format='ru' select /*+ agg_to_cop(), hash_agg() */ b, count(*) from explain_ru_cop group by b", + opClasses: []string{ + "tikv/agg_hash", + "tikv/kv_range_scan", + }, + }, + { + sql: "explain analyze format='ru' select /*+ agg_to_cop(), stream_agg() */ b, count(*) from explain_ru_cop group by b", + opClasses: []string{ + "tikv/agg_stream", + "tikv/kv_range_scan", + }, + }, + } + for _, tc := range cases { + rows := tk.MustQuery(tc.sql).Rows() + for _, opClass := range tc.opClasses { + requireExplainRUWeightedOperatorClass(t, rows, opClass) + } + } +} + +func requireExplainRUPlanRow(t *testing.T, rows [][]any) { + t.Helper() + for _, row := range rows { + require.Len(t, row, 17) + if row[0] != "plan" { + continue + } + require.NotEmpty(t, row[1]) + require.NotEmpty(t, row[2]) + require.Contains(t, fmt.Sprint(row[3]), "/") + require.NotEmpty(t, row[4]) + require.NotEmpty(t, row[7]) + require.NotEmpty(t, row[8]) + require.NotEmpty(t, row[11]) + require.NotEmpty(t, row[12]) + require.NotEmpty(t, row[13]) + require.NotEmpty(t, row[14]) + require.NotEmpty(t, row[15]) + require.Contains(t, fmt.Sprint(row[16]), "weight_version=v1") + return + } + require.Fail(t, "missing FORMAT='RU' plan row") +} + +func requireExplainRUOperatorClass(t *testing.T, rows [][]any, operatorClass string) { + t.Helper() + for _, row := range rows { + require.Len(t, row, 17) + if row[0] != "plan" || row[3] != operatorClass { + continue + } + return + } + require.Failf(t, "missing FORMAT='RU' operator class", "operatorClass=%s rows=%v", operatorClass, rows) +} + +func requireExplainRUWeightedOperatorClass(t *testing.T, rows [][]any, operatorClass string) { + t.Helper() + for _, row := range rows { + require.Len(t, row, 17) + if row[0] != "plan" || row[3] != operatorClass { + continue + } + require.NotEmpty(t, row[13], "missing weight for %s row %v", operatorClass, row) + require.NotEmpty(t, row[14], "missing preview RU for %s row %v", operatorClass, row) + require.Contains(t, fmt.Sprint(row[16]), "weight_version=v1") + return + } + require.Failf(t, "missing weighted FORMAT='RU' operator class", "operatorClass=%s rows=%v", operatorClass, rows) +} + +func TestExplainAnalyzeFormatRUPlanDigest(t *testing.T) { + store := testkit.CreateMockStore(t) + tk := testkit.NewTestKit(t, store) + require.NoError(t, tk.Session().Auth(&auth.UserIdentity{Username: "root", Hostname: "%"}, nil, nil, nil)) + tk.MustExec("use test") + + originEnableStmtSummary := fmt.Sprint(tk.MustQuery("select @@global.tidb_enable_stmt_summary").Rows()[0][0]) + originStmtSummaryRefreshInterval := fmt.Sprint(tk.MustQuery("select @@global.tidb_stmt_summary_refresh_interval").Rows()[0][0]) + originStmtSummaryHistorySize := fmt.Sprint(tk.MustQuery("select @@global.tidb_stmt_summary_history_size").Rows()[0][0]) + defer tk.MustExec(fmt.Sprintf("set global tidb_enable_stmt_summary = %s", originEnableStmtSummary)) + defer tk.MustExec(fmt.Sprintf("set global tidb_stmt_summary_refresh_interval = %s", originStmtSummaryRefreshInterval)) + defer tk.MustExec(fmt.Sprintf("set global tidb_stmt_summary_history_size = %s", originStmtSummaryHistorySize)) + tk.MustExec("set global tidb_stmt_summary_history_size = 24") + tk.MustExec("set global tidb_stmt_summary_refresh_interval = 999999999") + tk.MustExec("set global tidb_enable_stmt_summary = 0") + tk.MustExec("set global tidb_enable_stmt_summary = 1") + + tk.MustExec("drop table if exists explain_ru_digest") + tk.MustExec("create table explain_ru_digest(a int primary key, b int)") + tk.MustExec("insert into explain_ru_digest values (1, 10)") + tk.MustQuery("select b from explain_ru_digest where a = 1").Check(testkit.Rows("10")) + + digestRows := tk.MustQuery("select plan_digest from information_schema.statements_summary_history where digest_text like 'select `b` from `explain_ru_digest`%' and plan_digest != ''").Rows() + require.NotEmpty(t, digestRows) + planDigest := fmt.Sprint(digestRows[0][0]) + rows := tk.MustQuery(fmt.Sprintf("explain analyze format='ru' '%s'", planDigest)).Rows() + require.NotEmpty(t, rows) + require.Equal(t, "summary", rows[0][0]) + requireExplainRUPlanRow(t, rows) +} + +func TestExplainAnalyzeFormatRUUnsupportedTargetsBeforeExecution(t *testing.T) { + store := testkit.CreateMockStore(t) + tk := testkit.NewTestKit(t, store) + tk.MustExec("use test") + tk.MustExec("drop table if exists explain_ru_dml") + tk.MustExec("create table explain_ru_dml(a int primary key)") + tk.MustExec("insert into explain_ru_dml values (1)") + + err := tk.ExecToErr("explain format='ru' select 1") + require.Error(t, err) + require.Contains(t, err.Error(), "cannot work without 'analyze'") + require.Contains(t, err.Error(), "format=ru") + + err = tk.ExecToErr("explain analyze format=ru select 1") + require.Error(t, err) + + err = tk.ExecToErr("explain analyze format='ru' insert into explain_ru_dml values (2)") + require.Error(t, err) + require.Contains(t, err.Error(), "unsupported_non_select") + tk.MustQuery("select * from explain_ru_dml order by a").Check(testkit.Rows("1")) + + for _, sql := range []string{ + "explain analyze format='ru' update explain_ru_dml set a = 2 where a = 1", + "explain analyze format='ru' delete from explain_ru_dml where a = 1", + "explain analyze format='ru' replace into explain_ru_dml values (2)", + "explain analyze format='ru' alter table explain_ru_dml add column b int", + "explain analyze format='ru' import into explain_ru_dml from select * from explain_ru_dml", + } { + err = tk.ExecToErr(sql) + require.Error(t, err, sql) + require.Contains(t, err.Error(), "unsupported_non_select", sql) + tk.MustQuery("select * from explain_ru_dml order by a").Check(testkit.Rows("1")) + } + tk.MustQuery("show columns from explain_ru_dml").Check(testkit.Rows("a int(11) NO PRI ")) + + err = tk.ExecToErr("explain analyze format='ru' values (1)") + require.Error(t, err) + + err = tk.ExecToErr("explain analyze format='ru' table explain_ru_dml") + require.Error(t, err) + require.Contains(t, err.Error(), "unsupported_non_select") + + tk.MustExec("set @explain_ru_var := 7") + err = tk.ExecToErr("explain analyze format='ru' select @explain_ru_var := 9") + require.Error(t, err) + require.Contains(t, err.Error(), "unsupported_side_effecting_select") + tk.MustQuery("select @explain_ru_var").Check(testkit.Rows("7")) + + tk.MustQuery("select last_insert_id(11)").Check(testkit.Rows("11")) + err = tk.ExecToErr("explain analyze format='ru' select last_insert_id(123)") + require.Error(t, err) + require.Contains(t, err.Error(), "unsupported_side_effecting_select") + tk.MustQuery("select last_insert_id()").Check(testkit.Rows("11")) + + outFile := filepath.Join(t.TempDir(), "explain_ru.csv") + err = tk.ExecToErr(fmt.Sprintf("explain analyze format='ru' select 1 into outfile %q", outFile)) + require.Error(t, err) + require.Contains(t, err.Error(), "unsupported_side_effecting_select") + _, statErr := os.Stat(outFile) + require.True(t, os.IsNotExist(statErr)) + + err = tk.ExecToErr("explain analyze format='ru' select * from explain_ru_dml for update skip locked") + require.Error(t, err) + require.Contains(t, err.Error(), "unsupported_locking_select") + + err = tk.ExecToErr("explain analyze format='ru' select get_lock('explain_ru', 0)") + require.Error(t, err) + require.Contains(t, err.Error(), "unsupported_side_effecting_select") + tk.MustQuery("select is_free_lock('explain_ru')").Check(testkit.Rows("1")) +} + +func TestReadBillingDemoMetricsHook(t *testing.T) { + metrics.InitExplainRUMetrics() + + store := testkit.CreateMockStore(t) + tk := testkit.NewTestKit(t, store) + require.NoError(t, tk.Session().Auth(&auth.UserIdentity{Username: "root", Hostname: "%"}, nil, nil, nil)) + tk.MustExec("use test") + + originEnableStmtSummary := fmt.Sprint(tk.MustQuery("select @@global.tidb_enable_stmt_summary").Rows()[0][0]) + defer tk.MustExec(fmt.Sprintf("set global tidb_enable_stmt_summary = %s", originEnableStmtSummary)) + tk.MustExec("set global tidb_enable_stmt_summary = 0") + tk.MustExec("set global tidb_enable_stmt_summary = 1") + + tk.MustQuery("select @@tidb_enable_read_billing_demo").Check(testkit.Rows("0")) + tk.MustExec("drop table if exists read_billing_demo") + tk.MustExec("create table read_billing_demo(a int primary key)") + tk.MustExec("insert into read_billing_demo values (1), (2)") + + success := metrics.ReadBillingDemoStatementsCounter.WithLabelValues("success", "v1") + unsupported := metrics.ReadBillingDemoStatementsCounter.WithLabelValues("unsupported", "v1") + unknownInput := metrics.ReadBillingDemoStatementsCounter.WithLabelValues("unknown_input", "v1") + errorStatus := metrics.ReadBillingDemoStatementsCounter.WithLabelValues("error", "v1") + projectionFixedEvents := metrics.ReadBillingDemoBaseUnitsCounter.WithLabelValues("tidb", "projection_eval", "projection", "fixed_events", "runtime_act_rows", "all", "v1") + + tk.MustQuery("select 1 + 1").Check(testkit.Rows("2")) + require.Equal(t, 0.0, readExecutorCounterValue(t, success)) + + tk.MustExec("set tidb_enable_read_billing_demo=on") + tk.MustQuery("select 1 + 1").Check(testkit.Rows("2")) + require.Equal(t, 1.0, readExecutorCounterValue(t, success)) + require.Equal(t, 1.0, readExecutorCounterValue(t, projectionFixedEvents)) + tk.MustExec("set tidb_enable_read_billing_demo=off") + tk.MustQuery(`select exec_count, sum_read_billing_demo_fixed_events > 0, sum_read_billing_demo_input_rows > 0, sum_read_billing_demo_input_bytes > 0 from information_schema.statements_summary where digest_text = 'select ? + ?'`).Check(testkit.Rows("2 1 1 1")) + tk.MustQuery(`select site, op_class, operator_kind, unit, input_source, input_side, model_version, weight_version, sample_count, value > 0, avg_row_width > 0 from information_schema.statements_summary_read_billing_demo_base_units where digest_text = 'select ? + ?' and site = 'tidb' and op_class = 'projection_eval' and operator_kind = 'projection' and unit = 'fixed_events'`).Check(testkit.Rows("tidb projection_eval projection fixed_events runtime_act_rows all v1 v1 1 1 1")) + tk.MustQuery(`select site, op_class, operator_kind, status, reason, count from information_schema.statements_summary_read_billing_demo_status where digest_text = 'select ? + ?' and site = 'statement'`).Check(testkit.Rows("statement statement statement success none 1")) + tk.MustQuery(`select column_name from information_schema.columns where table_schema = 'INFORMATION_SCHEMA' and table_name = 'CLUSTER_STATEMENTS_SUMMARY_READ_BILLING_DEMO_BASE_UNITS' and ordinal_position = 1`).Check(testkit.Rows("INSTANCE")) + tk.MustExec("set tidb_enable_read_billing_demo=on") + + beforeRestrictedSuccess := readExecutorCounterValue(t, success) + beforeRestrictedBaseUnits := readExecutorCounterValue(t, projectionFixedEvents) + internalCtx := kv.WithInternalSourceType(context.Background(), kv.InternalTxnOthers) + _, _, restrictedErr := tk.Session().GetRestrictedSQLExecutor().ExecRestrictedSQL(internalCtx, []sqlexec.OptionFuncAlias{sqlexec.ExecOptionUseCurSession}, "select 1 + 1") + require.NoError(t, restrictedErr) + require.Equal(t, beforeRestrictedSuccess, readExecutorCounterValue(t, success)) + require.Equal(t, beforeRestrictedBaseUnits, readExecutorCounterValue(t, projectionFixedEvents)) + + tk.MustExec("set tidb_enable_read_billing_demo=off") + tk.MustExec("prepare read_billing_demo_stmt from 'select ? + 1'") + tk.MustExec("set @read_billing_demo_param := 2") + tk.MustExec("set tidb_enable_read_billing_demo=on") + tk.MustQuery("execute read_billing_demo_stmt using @read_billing_demo_param").Check(testkit.Rows("3")) + require.Equal(t, 2.0, readExecutorCounterValue(t, success)) + + beforeBaseUnits := readExecutorCounterValue(t, projectionFixedEvents) + beforeBaseUnitsTotal := readExecutorCounterVecTotal(t, metrics.ReadBillingDemoBaseUnitsCounter) + beforeUnsupported := readExecutorCounterValue(t, unsupported) + tk.MustExec("insert into read_billing_demo values (3)") + require.Equal(t, beforeUnsupported+1, readExecutorCounterValue(t, unsupported)) + require.Equal(t, beforeBaseUnits, readExecutorCounterValue(t, projectionFixedEvents)) + require.Equal(t, beforeBaseUnitsTotal, readExecutorCounterVecTotal(t, metrics.ReadBillingDemoBaseUnitsCounter)) + tk.MustQuery("select status, reason, sum(count) from information_schema.statements_summary_read_billing_demo_status where digest_text like 'insert into `read_billing_demo`%' and status = 'unsupported' group by status, reason").Check(testkit.Rows("unsupported unsupported_non_select 1")) + + beforeUnknownInput := readExecutorCounterValue(t, unknownInput) + beforeBaseUnits = readExecutorCounterValue(t, projectionFixedEvents) + beforeBaseUnitsTotal = readExecutorCounterVecTotal(t, metrics.ReadBillingDemoBaseUnitsCounter) + plannercore.RecordReadBillingDemoForStatement(tk.Session(), nil, nil, nil) + require.Equal(t, beforeUnknownInput+1, readExecutorCounterValue(t, unknownInput)) + require.Equal(t, beforeBaseUnits, readExecutorCounterValue(t, projectionFixedEvents)) + require.Equal(t, beforeBaseUnitsTotal, readExecutorCounterVecTotal(t, metrics.ReadBillingDemoBaseUnitsCounter)) + + tk.MustExec("set tidb_enable_read_billing_demo=off") + tk.MustExec("create table read_billing_compile_error(a int)") + tk.MustExec("prepare read_billing_compile_error_stmt from 'select * from read_billing_compile_error'") + tk.MustExec("drop table read_billing_compile_error") + tk.MustExec("set tidb_enable_read_billing_demo=on") + beforeError := readExecutorCounterValue(t, errorStatus) + beforeBaseUnitsTotal = readExecutorCounterVecTotal(t, metrics.ReadBillingDemoBaseUnitsCounter) + err := tk.ExecToErr("execute read_billing_compile_error_stmt") + require.Error(t, err) + require.Equal(t, beforeError+1, readExecutorCounterValue(t, errorStatus)) + require.Equal(t, beforeBaseUnitsTotal, readExecutorCounterVecTotal(t, metrics.ReadBillingDemoBaseUnitsCounter)) + + beforeEarlyError := readExecutorCounterValue(t, errorStatus) + canceledCtx, cancel := context.WithCancel(context.Background()) + cancel() + _, err = tk.ExecWithContext(canceledCtx, "select 1") + require.Error(t, err) + require.Equal(t, beforeEarlyError+1, readExecutorCounterValue(t, errorStatus)) + require.Equal(t, beforeBaseUnitsTotal, readExecutorCounterVecTotal(t, metrics.ReadBillingDemoBaseUnitsCounter)) + + tk.MustExec("drop table if exists read_billing_outer") + tk.MustExec("create table read_billing_outer(a int)") + tk.MustExec("insert into read_billing_outer values (0)") + tk.MustExec("set @@tidb_init_chunk_size=1") + rs, err := tk.Exec("select (select t.a from read_billing_demo t where t.a > o.a) from read_billing_outer o") + require.NoError(t, err) + err = rs.Next(context.TODO(), rs.NewChunk(nil)) + require.Error(t, err) + require.NoError(t, rs.Close()) + require.Equal(t, beforeEarlyError+2, readExecutorCounterValue(t, errorStatus)) + require.Equal(t, beforeBaseUnitsTotal, readExecutorCounterVecTotal(t, metrics.ReadBillingDemoBaseUnitsCounter)) + tk.MustQuery(`select sum(count) from information_schema.statements_summary_read_billing_demo_status where site = 'statement' and status = 'error' and reason = 'statement_error'`).Check(testkit.Rows("3")) + tk.MustQuery(`select count(*) from information_schema.statements_summary_read_billing_demo_base_units b join information_schema.statements_summary_read_billing_demo_status s on b.digest = s.digest where s.status in ('unsupported', 'error')`).Check(testkit.Rows("0")) +} + +func readExecutorCounterValue(t *testing.T, counter prometheus.Counter) float64 { + t.Helper() + m := &dto.Metric{} + require.NoError(t, counter.Write(m)) + return m.GetCounter().GetValue() +} + +func readExecutorCounterVecTotal(t *testing.T, collector prometheus.Collector) float64 { + t.Helper() + ch := make(chan prometheus.Metric) + go func() { + collector.Collect(ch) + close(ch) + }() + var total float64 + for metric := range ch { + m := &dto.Metric{} + require.NoError(t, metric.Write(m)) + total += m.GetCounter().GetValue() + } + return total +} + func TestExplainImportFromSelect(t *testing.T) { store := testkit.CreateMockStore(t) tk := testkit.NewTestKit(t, store) diff --git a/pkg/executor/explainfor_test.go b/pkg/executor/explainfor_test.go index d2238ae879c60..1d25cdd26c045 100644 --- a/pkg/executor/explainfor_test.go +++ b/pkg/executor/explainfor_test.go @@ -336,6 +336,28 @@ func TestExplainDotForExplainPlan(t *testing.T) { require.Contains(t, err.Error(), "explain format 'dot' for connection is not supported now") } +func TestExplainRUForConnectionUnsupported(t *testing.T) { + store := testkit.CreateMockStore(t) + tk := testkit.NewTestKit(t, store) + + connIDRows := tk.MustQuery("select connection_id()").Rows() + require.Len(t, connIDRows, 1) + connID := connIDRows[0][0].(string) + + tkProcess := tk.Session().ShowProcess() + tk.Session().SetSessionManager(&testkit.MockSessionManager{PS: []*sessmgr.ProcessInfo{tkProcess}}) + err := tk.ExecToErr(fmt.Sprintf("explain format='ru' for connection %s", connID)) + require.Error(t, err) + require.Contains(t, err.Error(), "unsupported_for_connection") + + tk.MustQuery("select 1") + tkProcess = tk.Session().ShowProcess() + tk.Session().SetSessionManager(&testkit.MockSessionManager{PS: []*sessmgr.ProcessInfo{tkProcess}}) + err = tk.ExecToErr(fmt.Sprintf("explain format='ru' for connection %s", connID)) + require.Error(t, err) + require.Contains(t, err.Error(), "unsupported_for_connection") +} + func TestExplainDotForQuery(t *testing.T) { store := testkit.CreateMockStore(t) tk := testkit.NewTestKit(t, store) diff --git a/pkg/executor/stmtsummary.go b/pkg/executor/stmtsummary.go index 990a1808a66f3..12c2b7a71115e 100644 --- a/pkg/executor/stmtsummary.go +++ b/pkg/executor/stmtsummary.go @@ -159,7 +159,13 @@ func (e *stmtSummaryRetriever) initSummaryRowsReader(sctx sessionctx.Context) (* } var rows [][]types.Datum - if isCumulativeTable(e.table.Name.O) { + if kind, ok := readBillingDemoTableKind(e.table.Name.O); ok { + if isCurrentTable(e.table.Name.O) { + rows = reader.GetReadBillingDemoCurrentRows(kind) + } else if isHistoryTable(e.table.Name.O) { + rows = reader.GetReadBillingDemoHistoryRows(kind) + } + } else if isCumulativeTable(e.table.Name.O) { rows = reader.GetStmtSummaryCumulativeRows() } else if isCurrentTable(e.table.Name.O) { rows = reader.GetStmtSummaryCurrentRows() @@ -254,8 +260,14 @@ func (r *stmtSummaryRetrieverV2) initSummaryRowsReader(ctx context.Context, sctx return nil, err } - mem := stmtsummaryv2.NewMemReader(stmtSummary, columns, instanceAddr, tz, user, priv, digests, timeRanges) - memRows := mem.Rows() + var memRows [][]types.Datum + var readBillingDemoKind stmtsummary.ReadBillingDemoTableKind + readBillingDemoKind, isReadBillingDemoTable := readBillingDemoTableKind(r.table.Name.O) + if isReadBillingDemoTable { + memRows = stmtsummaryv2.NewReadBillingDemoMemReader(stmtSummary, columns, instanceAddr, tz, user, priv, digests, timeRanges, readBillingDemoKind).Rows() + } else { + memRows = stmtsummaryv2.NewMemReader(stmtSummary, columns, instanceAddr, tz, user, priv, digests, timeRanges).Rows() + } var rowsReader *rowsReader if isCurrentTable(r.table.Name.O) { @@ -264,7 +276,13 @@ func (r *stmtSummaryRetrieverV2) initSummaryRowsReader(ctx context.Context, sctx if isHistoryTable(r.table.Name.O) { // history table should return all rows including mem and disk concurrent := sctx.GetSessionVars().Concurrency.DistSQLScanConcurrency() - history, err := stmtsummaryv2.NewHistoryReader(ctx, columns, instanceAddr, tz, user, priv, digests, timeRanges, concurrent) + var history rowsPuller + var err error + if isReadBillingDemoTable { + history, err = stmtsummaryv2.NewReadBillingDemoHistoryReader(ctx, columns, instanceAddr, tz, user, priv, digests, timeRanges, concurrent, readBillingDemoKind) + } else { + history, err = stmtsummaryv2.NewHistoryReader(ctx, columns, instanceAddr, tz, user, priv, digests, timeRanges, concurrent) + } if err != nil { return nil, err } @@ -347,6 +365,10 @@ func isClusterTable(originalTableName string) bool { case infoschema.ClusterTableStatementsSummary, infoschema.ClusterTableStatementsSummaryHistory, infoschema.ClusterTableStatementsSummaryEvicted, + infoschema.ClusterTableStatementsSummaryReadBillingDemoBaseUnits, + infoschema.ClusterTableStatementsSummaryHistoryReadBillingDemoBaseUnits, + infoschema.ClusterTableStatementsSummaryReadBillingDemoStatus, + infoschema.ClusterTableStatementsSummaryHistoryReadBillingDemoStatus, infoschema.ClusterTableTiDBStatementsStats: return true } @@ -367,7 +389,11 @@ func isCumulativeTable(originalTableName string) bool { func isCurrentTable(originalTableName string) bool { switch originalTableName { case infoschema.TableStatementsSummary, - infoschema.ClusterTableStatementsSummary: + infoschema.ClusterTableStatementsSummary, + infoschema.TableStatementsSummaryReadBillingDemoBaseUnits, + infoschema.ClusterTableStatementsSummaryReadBillingDemoBaseUnits, + infoschema.TableStatementsSummaryReadBillingDemoStatus, + infoschema.ClusterTableStatementsSummaryReadBillingDemoStatus: return true } @@ -377,13 +403,34 @@ func isCurrentTable(originalTableName string) bool { func isHistoryTable(originalTableName string) bool { switch originalTableName { case infoschema.TableStatementsSummaryHistory, - infoschema.ClusterTableStatementsSummaryHistory: + infoschema.ClusterTableStatementsSummaryHistory, + infoschema.TableStatementsSummaryHistoryReadBillingDemoBaseUnits, + infoschema.ClusterTableStatementsSummaryHistoryReadBillingDemoBaseUnits, + infoschema.TableStatementsSummaryHistoryReadBillingDemoStatus, + infoschema.ClusterTableStatementsSummaryHistoryReadBillingDemoStatus: return true } return false } +func readBillingDemoTableKind(originalTableName string) (stmtsummary.ReadBillingDemoTableKind, bool) { + switch originalTableName { + case infoschema.TableStatementsSummaryReadBillingDemoBaseUnits, + infoschema.TableStatementsSummaryHistoryReadBillingDemoBaseUnits, + infoschema.ClusterTableStatementsSummaryReadBillingDemoBaseUnits, + infoschema.ClusterTableStatementsSummaryHistoryReadBillingDemoBaseUnits: + return stmtsummary.ReadBillingDemoTableBaseUnits, true + case infoschema.TableStatementsSummaryReadBillingDemoStatus, + infoschema.TableStatementsSummaryHistoryReadBillingDemoStatus, + infoschema.ClusterTableStatementsSummaryReadBillingDemoStatus, + infoschema.ClusterTableStatementsSummaryHistoryReadBillingDemoStatus: + return stmtsummary.ReadBillingDemoTableStatus, true + default: + return 0, false + } +} + func isEvictedTable(originalTableName string) bool { switch originalTableName { case infoschema.TableStatementsSummaryEvicted, diff --git a/pkg/infoschema/cluster.go b/pkg/infoschema/cluster.go index 6ed27e538af87..9bc4934a9ccff 100644 --- a/pkg/infoschema/cluster.go +++ b/pkg/infoschema/cluster.go @@ -43,6 +43,14 @@ const ( ClusterTableStatementsSummaryHistory = "CLUSTER_STATEMENTS_SUMMARY_HISTORY" // ClusterTableStatementsSummaryEvicted is the string constant of cluster statement summary evict table. ClusterTableStatementsSummaryEvicted = "CLUSTER_STATEMENTS_SUMMARY_EVICTED" + // ClusterTableStatementsSummaryReadBillingDemoBaseUnits is the string constant of cluster read billing demo base-unit summary table. + ClusterTableStatementsSummaryReadBillingDemoBaseUnits = "CLUSTER_STATEMENTS_SUMMARY_READ_BILLING_DEMO_BASE_UNITS" + // ClusterTableStatementsSummaryHistoryReadBillingDemoBaseUnits is the string constant of cluster read billing demo history base-unit summary table. + ClusterTableStatementsSummaryHistoryReadBillingDemoBaseUnits = "CLUSTER_STATEMENTS_SUMMARY_HISTORY_READ_BILLING_DEMO_BASE_UNITS" + // ClusterTableStatementsSummaryReadBillingDemoStatus is the string constant of cluster read billing demo status summary table. + ClusterTableStatementsSummaryReadBillingDemoStatus = "CLUSTER_STATEMENTS_SUMMARY_READ_BILLING_DEMO_STATUS" + // ClusterTableStatementsSummaryHistoryReadBillingDemoStatus is the string constant of cluster read billing demo history status summary table. + ClusterTableStatementsSummaryHistoryReadBillingDemoStatus = "CLUSTER_STATEMENTS_SUMMARY_HISTORY_READ_BILLING_DEMO_STATUS" // ClusterTableTiDBStatementsStats is the string constant of the cluster statement stats table. ClusterTableTiDBStatementsStats = "CLUSTER_TIDB_STATEMENTS_STATS" // ClusterTableTiDBTrx is the string constant of cluster transaction running table. @@ -63,19 +71,23 @@ const ( // memTableToAllTiDBClusterTables means add memory table to cluster table that will send cop request to all TiDB nodes. var memTableToAllTiDBClusterTables = map[string]string{ - TableSlowQuery: ClusterTableSlowLog, - TableProcesslist: ClusterTableProcesslist, - TableStatementsSummary: ClusterTableStatementsSummary, - TableStatementsSummaryHistory: ClusterTableStatementsSummaryHistory, - TableStatementsSummaryEvicted: ClusterTableStatementsSummaryEvicted, - TableTiDBStatementsStats: ClusterTableTiDBStatementsStats, - TableTiDBTrx: ClusterTableTiDBTrx, - TableDeadlocks: ClusterTableDeadlocks, - TableTrxSummary: ClusterTableTrxSummary, - TableMemoryUsage: ClusterTableMemoryUsage, - TableMemoryUsageOpsHistory: ClusterTableMemoryUsageOpsHistory, - TableTiDBIndexUsage: ClusterTableTiDBIndexUsage, - TableTiDBPlanCache: ClusterTableTiDBPlanCache, + TableSlowQuery: ClusterTableSlowLog, + TableProcesslist: ClusterTableProcesslist, + TableStatementsSummary: ClusterTableStatementsSummary, + TableStatementsSummaryHistory: ClusterTableStatementsSummaryHistory, + TableStatementsSummaryEvicted: ClusterTableStatementsSummaryEvicted, + TableStatementsSummaryReadBillingDemoBaseUnits: ClusterTableStatementsSummaryReadBillingDemoBaseUnits, + TableStatementsSummaryHistoryReadBillingDemoBaseUnits: ClusterTableStatementsSummaryHistoryReadBillingDemoBaseUnits, + TableStatementsSummaryReadBillingDemoStatus: ClusterTableStatementsSummaryReadBillingDemoStatus, + TableStatementsSummaryHistoryReadBillingDemoStatus: ClusterTableStatementsSummaryHistoryReadBillingDemoStatus, + TableTiDBStatementsStats: ClusterTableTiDBStatementsStats, + TableTiDBTrx: ClusterTableTiDBTrx, + TableDeadlocks: ClusterTableDeadlocks, + TableTrxSummary: ClusterTableTrxSummary, + TableMemoryUsage: ClusterTableMemoryUsage, + TableMemoryUsageOpsHistory: ClusterTableMemoryUsageOpsHistory, + TableTiDBIndexUsage: ClusterTableTiDBIndexUsage, + TableTiDBPlanCache: ClusterTableTiDBPlanCache, } var memTableToAllTiDBClusterTablesWithLowerCase = make(map[string]string) diff --git a/pkg/infoschema/tables.go b/pkg/infoschema/tables.go index 3072d29e07c87..a620816e06703 100644 --- a/pkg/infoschema/tables.go +++ b/pkg/infoschema/tables.go @@ -170,6 +170,14 @@ const ( TableStatementsSummaryHistory = "STATEMENTS_SUMMARY_HISTORY" // TableStatementsSummaryEvicted is the string constant of statements summary evicted table. TableStatementsSummaryEvicted = "STATEMENTS_SUMMARY_EVICTED" + // TableStatementsSummaryReadBillingDemoBaseUnits is the string constant of read billing demo base-unit summary table. + TableStatementsSummaryReadBillingDemoBaseUnits = "STATEMENTS_SUMMARY_READ_BILLING_DEMO_BASE_UNITS" + // TableStatementsSummaryHistoryReadBillingDemoBaseUnits is the string constant of read billing demo history base-unit summary table. + TableStatementsSummaryHistoryReadBillingDemoBaseUnits = "STATEMENTS_SUMMARY_HISTORY_READ_BILLING_DEMO_BASE_UNITS" + // TableStatementsSummaryReadBillingDemoStatus is the string constant of read billing demo status summary table. + TableStatementsSummaryReadBillingDemoStatus = "STATEMENTS_SUMMARY_READ_BILLING_DEMO_STATUS" + // TableStatementsSummaryHistoryReadBillingDemoStatus is the string constant of read billing demo history status summary table. + TableStatementsSummaryHistoryReadBillingDemoStatus = "STATEMENTS_SUMMARY_HISTORY_READ_BILLING_DEMO_STATUS" // TableTiDBStatementsStats is the string constant of the TiDB statement stats table. TableTiDBStatementsStats = "TIDB_STATEMENTS_STATS" // TableStorageStats is a table that contains all tables disk usage @@ -319,41 +327,49 @@ var tableIDMap = map[string]int64{ TableTiFlashSegments: autoid.InformationSchemaDBID + 65, // Removed, see https://github.com/pingcap/tidb/issues/28890 //TablePlacementPolicy: autoid.InformationSchemaDBID + 66, - TableClientErrorsSummaryGlobal: autoid.InformationSchemaDBID + 67, - TableClientErrorsSummaryByUser: autoid.InformationSchemaDBID + 68, - TableClientErrorsSummaryByHost: autoid.InformationSchemaDBID + 69, - TableTiDBTrx: autoid.InformationSchemaDBID + 70, - ClusterTableTiDBTrx: autoid.InformationSchemaDBID + 71, - TableDeadlocks: autoid.InformationSchemaDBID + 72, - ClusterTableDeadlocks: autoid.InformationSchemaDBID + 73, - TableDataLockWaits: autoid.InformationSchemaDBID + 74, - TableStatementsSummaryEvicted: autoid.InformationSchemaDBID + 75, - ClusterTableStatementsSummaryEvicted: autoid.InformationSchemaDBID + 76, - TableAttributes: autoid.InformationSchemaDBID + 77, - TableTiDBHotRegionsHistory: autoid.InformationSchemaDBID + 78, - TablePlacementPolicies: autoid.InformationSchemaDBID + 79, - TableTrxSummary: autoid.InformationSchemaDBID + 80, - ClusterTableTrxSummary: autoid.InformationSchemaDBID + 81, - TableVariablesInfo: autoid.InformationSchemaDBID + 82, - TableUserAttributes: autoid.InformationSchemaDBID + 83, - TableMemoryUsage: autoid.InformationSchemaDBID + 84, - TableMemoryUsageOpsHistory: autoid.InformationSchemaDBID + 85, - ClusterTableMemoryUsage: autoid.InformationSchemaDBID + 86, - ClusterTableMemoryUsageOpsHistory: autoid.InformationSchemaDBID + 87, - TableResourceGroups: autoid.InformationSchemaDBID + 88, - TableRunawayWatches: autoid.InformationSchemaDBID + 89, - TableCheckConstraints: autoid.InformationSchemaDBID + 90, - TableTiDBCheckConstraints: autoid.InformationSchemaDBID + 91, - TableKeywords: autoid.InformationSchemaDBID + 92, - TableTiDBIndexUsage: autoid.InformationSchemaDBID + 93, - ClusterTableTiDBIndexUsage: autoid.InformationSchemaDBID + 94, - TableTiFlashIndexes: autoid.InformationSchemaDBID + 95, - TableTiDBPlanCache: autoid.InformationSchemaDBID + 96, - ClusterTableTiDBPlanCache: autoid.InformationSchemaDBID + 97, - TableTiDBStatementsStats: autoid.InformationSchemaDBID + 98, - ClusterTableTiDBStatementsStats: autoid.InformationSchemaDBID + 99, - TableKeyspaceMeta: autoid.InformationSchemaDBID + 100, - TableSchemataExtensions: autoid.InformationSchemaDBID + 101, + TableClientErrorsSummaryGlobal: autoid.InformationSchemaDBID + 67, + TableClientErrorsSummaryByUser: autoid.InformationSchemaDBID + 68, + TableClientErrorsSummaryByHost: autoid.InformationSchemaDBID + 69, + TableTiDBTrx: autoid.InformationSchemaDBID + 70, + ClusterTableTiDBTrx: autoid.InformationSchemaDBID + 71, + TableDeadlocks: autoid.InformationSchemaDBID + 72, + ClusterTableDeadlocks: autoid.InformationSchemaDBID + 73, + TableDataLockWaits: autoid.InformationSchemaDBID + 74, + TableStatementsSummaryEvicted: autoid.InformationSchemaDBID + 75, + ClusterTableStatementsSummaryEvicted: autoid.InformationSchemaDBID + 76, + TableAttributes: autoid.InformationSchemaDBID + 77, + TableTiDBHotRegionsHistory: autoid.InformationSchemaDBID + 78, + TablePlacementPolicies: autoid.InformationSchemaDBID + 79, + TableTrxSummary: autoid.InformationSchemaDBID + 80, + ClusterTableTrxSummary: autoid.InformationSchemaDBID + 81, + TableVariablesInfo: autoid.InformationSchemaDBID + 82, + TableUserAttributes: autoid.InformationSchemaDBID + 83, + TableMemoryUsage: autoid.InformationSchemaDBID + 84, + TableMemoryUsageOpsHistory: autoid.InformationSchemaDBID + 85, + ClusterTableMemoryUsage: autoid.InformationSchemaDBID + 86, + ClusterTableMemoryUsageOpsHistory: autoid.InformationSchemaDBID + 87, + TableResourceGroups: autoid.InformationSchemaDBID + 88, + TableRunawayWatches: autoid.InformationSchemaDBID + 89, + TableCheckConstraints: autoid.InformationSchemaDBID + 90, + TableTiDBCheckConstraints: autoid.InformationSchemaDBID + 91, + TableKeywords: autoid.InformationSchemaDBID + 92, + TableTiDBIndexUsage: autoid.InformationSchemaDBID + 93, + ClusterTableTiDBIndexUsage: autoid.InformationSchemaDBID + 94, + TableTiFlashIndexes: autoid.InformationSchemaDBID + 95, + TableTiDBPlanCache: autoid.InformationSchemaDBID + 96, + ClusterTableTiDBPlanCache: autoid.InformationSchemaDBID + 97, + TableTiDBStatementsStats: autoid.InformationSchemaDBID + 98, + ClusterTableTiDBStatementsStats: autoid.InformationSchemaDBID + 99, + TableKeyspaceMeta: autoid.InformationSchemaDBID + 100, + TableSchemataExtensions: autoid.InformationSchemaDBID + 101, + TableStatementsSummaryReadBillingDemoBaseUnits: autoid.InformationSchemaDBID + 102, + TableStatementsSummaryHistoryReadBillingDemoBaseUnits: autoid.InformationSchemaDBID + 103, + ClusterTableStatementsSummaryReadBillingDemoBaseUnits: autoid.InformationSchemaDBID + 104, + ClusterTableStatementsSummaryHistoryReadBillingDemoBaseUnits: autoid.InformationSchemaDBID + 105, + TableStatementsSummaryReadBillingDemoStatus: autoid.InformationSchemaDBID + 106, + TableStatementsSummaryHistoryReadBillingDemoStatus: autoid.InformationSchemaDBID + 107, + ClusterTableStatementsSummaryReadBillingDemoStatus: autoid.InformationSchemaDBID + 108, + ClusterTableStatementsSummaryHistoryReadBillingDemoStatus: autoid.InformationSchemaDBID + 109, } // columnInfo represents the basic column information of all kinds of INFORMATION_SCHEMA tables @@ -1429,6 +1445,9 @@ var tableStatementsSummaryCols = []columnInfo{ {name: stmtsummary.AvgQueuedRcTimeStr, tp: mysql.TypeLonglong, size: 22, flag: mysql.NotNullFlag | mysql.UnsignedFlag, comment: "Average time of waiting for available request-units"}, {name: stmtsummary.MaxRequestUnitV2Str, tp: mysql.TypeDouble, flag: mysql.NotNullFlag | mysql.UnsignedFlag, size: 22, comment: "Max request-unit v2 cost of these statements"}, {name: stmtsummary.AvgRequestUnitV2Str, tp: mysql.TypeDouble, flag: mysql.NotNullFlag | mysql.UnsignedFlag, size: 22, comment: "Average request-unit v2 cost of these statements"}, + {name: stmtsummary.SumReadBillingDemoFixedEventsStr, tp: mysql.TypeDouble, flag: mysql.NotNullFlag | mysql.UnsignedFlag, size: 22, comment: "Total read billing demo fixed-event base units"}, + {name: stmtsummary.SumReadBillingDemoInputRowsStr, tp: mysql.TypeDouble, flag: mysql.NotNullFlag | mysql.UnsignedFlag, size: 22, comment: "Total read billing demo input-row base units"}, + {name: stmtsummary.SumReadBillingDemoInputBytesStr, tp: mysql.TypeDouble, flag: mysql.NotNullFlag | mysql.UnsignedFlag, size: 22, comment: "Total read billing demo input-byte base units"}, {name: stmtsummary.ResourceGroupName, tp: mysql.TypeVarchar, size: 64, comment: "Bind resource group name"}, {name: stmtsummary.PlanCacheUnqualifiedStr, tp: mysql.TypeLonglong, size: 20, flag: mysql.NotNullFlag, comment: "The number of times that these statements are not supported by the plan cache"}, {name: stmtsummary.PlanCacheUnqualifiedLastReasonStr, tp: mysql.TypeBlob, size: types.UnspecifiedLength, comment: "The last reason why the statement is not supported by the plan cache"}, @@ -1444,6 +1463,49 @@ var tableStatementsSummaryCols = []columnInfo{ {name: stmtsummary.StorageMPPStr, tp: mysql.TypeTiny, size: 1, flag: mysql.NotNullFlag, comment: "Whether the last statement read data from TiFlash"}, } +var tableStatementsSummaryReadBillingDemoBaseUnitCols = []columnInfo{ + {name: stmtsummary.SummaryBeginTimeStr, tp: mysql.TypeTimestamp, size: 26, flag: mysql.NotNullFlag, comment: "Begin time of this summary"}, + {name: stmtsummary.SummaryEndTimeStr, tp: mysql.TypeTimestamp, size: 26, flag: mysql.NotNullFlag, comment: "End time of this summary"}, + {name: stmtsummary.StmtTypeStr, tp: mysql.TypeVarchar, size: 64, flag: mysql.NotNullFlag, comment: "Statement type"}, + {name: stmtsummary.SchemaNameStr, tp: mysql.TypeVarchar, size: 64, comment: "Current schema"}, + {name: stmtsummary.DigestStr, tp: mysql.TypeVarchar, size: 64}, + {name: stmtsummary.DigestTextStr, tp: mysql.TypeBlob, size: types.UnspecifiedLength, flag: mysql.NotNullFlag, comment: "Normalized statement"}, + {name: stmtsummary.PlanDigestStr, tp: mysql.TypeVarchar, size: 64, comment: "Digest of its execution plan"}, + {name: stmtsummary.ResourceGroupName, tp: mysql.TypeVarchar, size: 64, comment: "Bind resource group name"}, + {name: stmtsummary.ReadBillingDemoModelVersionStr, tp: mysql.TypeVarchar, size: 32, flag: mysql.NotNullFlag, comment: "Read billing demo model version"}, + {name: stmtsummary.ReadBillingDemoWeightVersionStr, tp: mysql.TypeVarchar, size: 32, flag: mysql.NotNullFlag, comment: "Read billing demo weight version"}, + {name: stmtsummary.ReadBillingDemoSiteStr, tp: mysql.TypeVarchar, size: 16, flag: mysql.NotNullFlag, comment: "Read billing execution site"}, + {name: stmtsummary.ReadBillingDemoOpClassStr, tp: mysql.TypeVarchar, size: 64, flag: mysql.NotNullFlag, comment: "Read billing operator class"}, + {name: stmtsummary.ReadBillingDemoOperatorKindStr, tp: mysql.TypeVarchar, size: 64, flag: mysql.NotNullFlag, comment: "Physical operator kind"}, + {name: stmtsummary.ReadBillingDemoUnitStr, tp: mysql.TypeVarchar, size: 32, flag: mysql.NotNullFlag, comment: "Coefficient-free base unit"}, + {name: stmtsummary.ReadBillingDemoInputSourceStr, tp: mysql.TypeVarchar, size: 32, flag: mysql.NotNullFlag, comment: "Runtime input evidence source"}, + {name: stmtsummary.ReadBillingDemoInputSideStr, tp: mysql.TypeVarchar, size: 16, flag: mysql.NotNullFlag, comment: "Operator input side"}, + {name: stmtsummary.ReadBillingDemoRowWidthSource, tp: mysql.TypeVarchar, size: 32, flag: mysql.NotNullFlag, comment: "Row-width evidence source"}, + {name: stmtsummary.ReadBillingDemoValueStr, tp: mysql.TypeDouble, size: 22, flag: mysql.NotNullFlag | mysql.UnsignedFlag, comment: "Aggregated base-unit value"}, + {name: stmtsummary.ReadBillingDemoSampleCountStr, tp: mysql.TypeLonglong, size: 20, flag: mysql.NotNullFlag | mysql.UnsignedFlag, comment: "Number of unit samples"}, + {name: stmtsummary.ReadBillingDemoRowWidthSumStr, tp: mysql.TypeDouble, size: 22, flag: mysql.NotNullFlag | mysql.UnsignedFlag, comment: "Sum of row-width samples"}, + {name: stmtsummary.ReadBillingDemoAvgRowWidthStr, tp: mysql.TypeDouble, size: 22, flag: mysql.NotNullFlag | mysql.UnsignedFlag, comment: "Average row-width sample"}, +} + +var tableStatementsSummaryReadBillingDemoStatusCols = []columnInfo{ + {name: stmtsummary.SummaryBeginTimeStr, tp: mysql.TypeTimestamp, size: 26, flag: mysql.NotNullFlag, comment: "Begin time of this summary"}, + {name: stmtsummary.SummaryEndTimeStr, tp: mysql.TypeTimestamp, size: 26, flag: mysql.NotNullFlag, comment: "End time of this summary"}, + {name: stmtsummary.StmtTypeStr, tp: mysql.TypeVarchar, size: 64, flag: mysql.NotNullFlag, comment: "Statement type"}, + {name: stmtsummary.SchemaNameStr, tp: mysql.TypeVarchar, size: 64, comment: "Current schema"}, + {name: stmtsummary.DigestStr, tp: mysql.TypeVarchar, size: 64}, + {name: stmtsummary.DigestTextStr, tp: mysql.TypeBlob, size: types.UnspecifiedLength, flag: mysql.NotNullFlag, comment: "Normalized statement"}, + {name: stmtsummary.PlanDigestStr, tp: mysql.TypeVarchar, size: 64, comment: "Digest of its execution plan"}, + {name: stmtsummary.ResourceGroupName, tp: mysql.TypeVarchar, size: 64, comment: "Bind resource group name"}, + {name: stmtsummary.ReadBillingDemoModelVersionStr, tp: mysql.TypeVarchar, size: 32, flag: mysql.NotNullFlag, comment: "Read billing demo model version"}, + {name: stmtsummary.ReadBillingDemoWeightVersionStr, tp: mysql.TypeVarchar, size: 32, flag: mysql.NotNullFlag, comment: "Read billing demo weight version"}, + {name: stmtsummary.ReadBillingDemoSiteStr, tp: mysql.TypeVarchar, size: 16, flag: mysql.NotNullFlag, comment: "Read billing execution site"}, + {name: stmtsummary.ReadBillingDemoOpClassStr, tp: mysql.TypeVarchar, size: 64, flag: mysql.NotNullFlag, comment: "Read billing operator class"}, + {name: stmtsummary.ReadBillingDemoOperatorKindStr, tp: mysql.TypeVarchar, size: 64, flag: mysql.NotNullFlag, comment: "Physical operator kind"}, + {name: stmtsummary.ReadBillingDemoStatusStr, tp: mysql.TypeVarchar, size: 32, flag: mysql.NotNullFlag, comment: "Read billing accounting status"}, + {name: stmtsummary.ReadBillingDemoReasonStr, tp: mysql.TypeVarchar, size: 128, flag: mysql.NotNullFlag, comment: "Read billing accounting status reason"}, + {name: stmtsummary.ReadBillingDemoCountStr, tp: mysql.TypeLonglong, size: 20, flag: mysql.NotNullFlag | mysql.UnsignedFlag, comment: "Number of status samples"}, +} + var tableTiDBStatementsStatsCols = []columnInfo{ {name: stmtsummary.StmtTypeStr, tp: mysql.TypeVarchar, size: 64, flag: mysql.NotNullFlag, comment: "Statement type"}, {name: stmtsummary.SchemaNameStr, tp: mysql.TypeVarchar, size: 64, comment: "Current schema"}, @@ -2516,32 +2578,36 @@ var tableNameToColumns = map[string][]columnInfo{ TableStatementsSummary: tableStatementsSummaryCols, TableStatementsSummaryHistory: tableStatementsSummaryCols, TableStatementsSummaryEvicted: tableStatementsSummaryEvictedCols, - TableStorageStats: tableStorageStatsCols, - TableTiDBStatementsStats: tableTiDBStatementsStatsCols, - TableTiFlashTables: tableTableTiFlashTablesCols, - TableTiFlashSegments: tableTableTiFlashSegmentsCols, - TableTiFlashIndexes: tableTiFlashIndexesCols, - TableClientErrorsSummaryGlobal: tableClientErrorsSummaryGlobalCols, - TableClientErrorsSummaryByUser: tableClientErrorsSummaryByUserCols, - TableClientErrorsSummaryByHost: tableClientErrorsSummaryByHostCols, - TableTiDBTrx: tableTiDBTrxCols, - TableDeadlocks: tableDeadlocksCols, - TableDataLockWaits: tableDataLockWaitsCols, - TableAttributes: tableAttributesCols, - TablePlacementPolicies: tablePlacementPoliciesCols, - TableTrxSummary: tableTrxSummaryCols, - TableVariablesInfo: tableVariablesInfoCols, - TableUserAttributes: tableUserAttributesCols, - TableMemoryUsage: tableMemoryUsageCols, - TableMemoryUsageOpsHistory: tableMemoryUsageOpsHistoryCols, - TableResourceGroups: tableResourceGroupsCols, - TableRunawayWatches: tableRunawayWatchListCols, - TableCheckConstraints: tableCheckConstraintsCols, - TableTiDBCheckConstraints: tableTiDBCheckConstraintsCols, - TableKeywords: tableKeywords, - TableTiDBIndexUsage: tableTiDBIndexUsage, - TableTiDBPlanCache: tablePlanCache, - TableKeyspaceMeta: tableKeyspaceMetaCols, + TableStatementsSummaryReadBillingDemoBaseUnits: tableStatementsSummaryReadBillingDemoBaseUnitCols, + TableStatementsSummaryHistoryReadBillingDemoBaseUnits: tableStatementsSummaryReadBillingDemoBaseUnitCols, + TableStatementsSummaryReadBillingDemoStatus: tableStatementsSummaryReadBillingDemoStatusCols, + TableStatementsSummaryHistoryReadBillingDemoStatus: tableStatementsSummaryReadBillingDemoStatusCols, + TableStorageStats: tableStorageStatsCols, + TableTiDBStatementsStats: tableTiDBStatementsStatsCols, + TableTiFlashTables: tableTableTiFlashTablesCols, + TableTiFlashSegments: tableTableTiFlashSegmentsCols, + TableTiFlashIndexes: tableTiFlashIndexesCols, + TableClientErrorsSummaryGlobal: tableClientErrorsSummaryGlobalCols, + TableClientErrorsSummaryByUser: tableClientErrorsSummaryByUserCols, + TableClientErrorsSummaryByHost: tableClientErrorsSummaryByHostCols, + TableTiDBTrx: tableTiDBTrxCols, + TableDeadlocks: tableDeadlocksCols, + TableDataLockWaits: tableDataLockWaitsCols, + TableAttributes: tableAttributesCols, + TablePlacementPolicies: tablePlacementPoliciesCols, + TableTrxSummary: tableTrxSummaryCols, + TableVariablesInfo: tableVariablesInfoCols, + TableUserAttributes: tableUserAttributesCols, + TableMemoryUsage: tableMemoryUsageCols, + TableMemoryUsageOpsHistory: tableMemoryUsageOpsHistoryCols, + TableResourceGroups: tableResourceGroupsCols, + TableRunawayWatches: tableRunawayWatchListCols, + TableCheckConstraints: tableCheckConstraintsCols, + TableTiDBCheckConstraints: tableTiDBCheckConstraintsCols, + TableKeywords: tableKeywords, + TableTiDBIndexUsage: tableTiDBIndexUsage, + TableTiDBPlanCache: tablePlanCache, + TableKeyspaceMeta: tableKeyspaceMetaCols, } func createInfoSchemaTable(_ autoid.Allocators, _ func() (pools.Resource, error), meta *model.TableInfo) (table.Table, error) { diff --git a/pkg/metrics/BUILD.bazel b/pkg/metrics/BUILD.bazel index 16b3aaa56e43e..44dba62738a42 100644 --- a/pkg/metrics/BUILD.bazel +++ b/pkg/metrics/BUILD.bazel @@ -9,6 +9,7 @@ go_library( "distsql.go", "domain.go", "executor.go", + "explain_ru.go", "external_workload.go", "gc_worker.go", "globalsort.go", @@ -69,7 +70,7 @@ go_test( ], embed = [":metrics"], flaky = True, - shard_count = 9, + shard_count = 11, deps = [ "//pkg/parser/terror", "//pkg/statistics/handle/cache", diff --git a/pkg/metrics/explain_ru.go b/pkg/metrics/explain_ru.go new file mode 100644 index 0000000000000..42d48e101beb7 --- /dev/null +++ b/pkg/metrics/explain_ru.go @@ -0,0 +1,217 @@ +// Copyright 2026 PingCAP, Inc. +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package metrics + +import ( + metricscommon "github.com/pingcap/tidb/pkg/metrics/common" + "github.com/prometheus/client_golang/prometheus" +) + +const ( + explainRULabelStatus = "status" + explainRULabelComponentSnapshotStatus = "component_snapshot_status" + readBillingDemoLabelModelVersion = "model_version" +) + +// EXPLAIN FORMAT='RU' demo metrics. +var ( + ExplainRUPreviewRUCounter *prometheus.CounterVec + ExplainRUWorkRowsCounter *prometheus.CounterVec + ExplainRUWorkBytesCounter *prometheus.CounterVec + ExplainRURowWidthHistogram *prometheus.HistogramVec + ExplainRUStatementsCounter *prometheus.CounterVec + ExplainRURenderDurationHistogram *prometheus.HistogramVec + ExplainRUComponentSnapshotCounter *prometheus.CounterVec + + ReadBillingDemoStatementsCounter *prometheus.CounterVec + ReadBillingDemoOperatorStatusCounter *prometheus.CounterVec + ReadBillingDemoBaseUnitsCounter *prometheus.CounterVec + ReadBillingDemoRowWidthHistogram *prometheus.HistogramVec +) + +// InitExplainRUMetrics initializes metrics for EXPLAIN ANALYZE FORMAT='RU'. +func InitExplainRUMetrics() { + // Keep these labels bounded during demo calibration. SQL text, digest, + // plan ID, table name, and index name belong in the SQL result, not metrics. + ExplainRUPreviewRUCounter = metricscommon.NewCounterVec( + prometheus.CounterOpts{ + Namespace: "tidb", + Subsystem: "explain_ru", + Name: "preview_ru_total", + Help: "Counter of read billing preview RU values generated by EXPLAIN ANALYZE FORMAT='RU'.", + }, []string{"section", "component", "operator", "source"}, + ) + ExplainRUWorkRowsCounter = metricscommon.NewCounterVec( + prometheus.CounterOpts{ + Namespace: "tidb", + Subsystem: "explain_ru", + Name: "work_rows_total", + Help: "Counter of row-count model inputs generated by EXPLAIN ANALYZE FORMAT='RU'.", + }, []string{"section", "component", "operator", "source"}, + ) + ExplainRUWorkBytesCounter = metricscommon.NewCounterVec( + prometheus.CounterOpts{ + Namespace: "tidb", + Subsystem: "explain_ru", + Name: "work_bytes_total", + Help: "Counter of byte-shaped model inputs generated by EXPLAIN ANALYZE FORMAT='RU'.", + }, []string{"section", "component", "operator", "source"}, + ) + ExplainRUStatementsCounter = metricscommon.NewCounterVec( + prometheus.CounterOpts{ + Namespace: "tidb", + Subsystem: "explain_ru", + Name: "statements_total", + Help: "Counter of EXPLAIN ANALYZE FORMAT='RU' statement status.", + }, []string{explainRULabelStatus}, + ) + ExplainRURenderDurationHistogram = metricscommon.NewHistogramVec( + prometheus.HistogramOpts{ + Namespace: "tidb", + Subsystem: "explain_ru", + Name: "render_duration_seconds", + Help: "Histogram of EXPLAIN ANALYZE FORMAT='RU' render duration.", + Buckets: prometheus.DefBuckets, + }, []string{explainRULabelStatus}, + ) + ExplainRURowWidthHistogram = metricscommon.NewHistogramVec( + prometheus.HistogramOpts{ + Namespace: "tidb", + Subsystem: "explain_ru", + Name: "row_width_bytes", + Help: "Histogram of row-width factors generated by EXPLAIN ANALYZE FORMAT='RU'.", + Buckets: prometheus.ExponentialBuckets(1, 2, 12), + }, []string{"component", "operator", "source"}, + ) + ExplainRUComponentSnapshotCounter = metricscommon.NewCounterVec( + prometheus.CounterOpts{ + Namespace: "tidb", + Subsystem: "explain_ru", + Name: "component_snapshot_total", + Help: "Counter of EXPLAIN ANALYZE FORMAT='RU' component snapshot status.", + }, []string{explainRULabelComponentSnapshotStatus}, + ) + ReadBillingDemoStatementsCounter = metricscommon.NewCounterVec( + prometheus.CounterOpts{ + Namespace: "tidb", + Subsystem: "read_billing_demo", + Name: "statements_total", + Help: "Counter of read billing demo statement status.", + }, []string{"status", readBillingDemoLabelModelVersion}, + ) + ReadBillingDemoOperatorStatusCounter = metricscommon.NewCounterVec( + prometheus.CounterOpts{ + Namespace: "tidb", + Subsystem: "read_billing_demo", + Name: "operator_status_total", + Help: "Counter of read billing demo operator status.", + }, []string{"site", "op_class", "operator_kind", "status", "reason", readBillingDemoLabelModelVersion}, + ) + ReadBillingDemoBaseUnitsCounter = metricscommon.NewCounterVec( + prometheus.CounterOpts{ + Namespace: "tidb", + Subsystem: "read_billing_demo", + Name: "base_units_total", + Help: "Counter of coefficient-free read billing demo base units.", + }, []string{"site", "op_class", "operator_kind", "unit", "input_source", "input_side", readBillingDemoLabelModelVersion}, + ) + ReadBillingDemoRowWidthHistogram = metricscommon.NewHistogramVec( + prometheus.HistogramOpts{ + Namespace: "tidb", + Subsystem: "read_billing_demo", + Name: "row_width_bytes", + Help: "Histogram of row-width factors used by read billing demo.", + Buckets: prometheus.ExponentialBuckets(1, 2, 12), + }, []string{"site", "op_class", "operator_kind", "row_width_source", readBillingDemoLabelModelVersion}, + ) +} + +// RecordExplainRUStatus records a bounded FORMAT='RU' statement status. +func RecordExplainRUStatus(status string) { + if ExplainRUStatementsCounter == nil || status == "" { + return + } + ExplainRUStatementsCounter.WithLabelValues(status).Inc() +} + +// ObserveExplainRURenderDuration records a bounded FORMAT='RU' render duration. +func ObserveExplainRURenderDuration(status string, seconds float64) { + if ExplainRURenderDurationHistogram == nil || status == "" || seconds < 0 { + return + } + ExplainRURenderDurationHistogram.WithLabelValues(status).Observe(seconds) +} + +// RecordExplainRUComponentSnapshot records component snapshot availability. +func RecordExplainRUComponentSnapshot(status string) { + if ExplainRUComponentSnapshotCounter == nil || status == "" { + return + } + ExplainRUComponentSnapshotCounter.WithLabelValues(status).Inc() +} + +// ObserveExplainRURow records bounded numeric samples from generated FORMAT='RU' rows. +func ObserveExplainRURow(section, component, operator, source, rowWidthSource string, previewRU, workRows, workBytes, rowWidth float64) { + // Negative values are sentinels for "not present" so callers can use one + // path for summary and plan rows without emitting fake zeros. + if ExplainRUPreviewRUCounter != nil && previewRU >= 0 { + ExplainRUPreviewRUCounter.WithLabelValues(section, component, operator, source).Add(previewRU) + } + if ExplainRUWorkRowsCounter != nil && workRows >= 0 { + ExplainRUWorkRowsCounter.WithLabelValues(section, component, operator, source).Add(workRows) + } + if ExplainRUWorkBytesCounter != nil && workBytes >= 0 { + ExplainRUWorkBytesCounter.WithLabelValues(section, component, operator, source).Add(workBytes) + } + if ExplainRURowWidthHistogram != nil && section == "plan" && rowWidth > 0 && operator != "" { + ExplainRURowWidthHistogram.WithLabelValues(component, operator, rowWidthSource).Observe(rowWidth) + } +} + +// RecordReadBillingDemoStatement records a bounded read billing demo statement status. +func RecordReadBillingDemoStatement(status, modelVersion string) { + if ReadBillingDemoStatementsCounter == nil || status == "" || modelVersion == "" { + return + } + ReadBillingDemoStatementsCounter.WithLabelValues(status, modelVersion).Inc() +} + +// RecordReadBillingDemoOperatorStatus records a bounded read billing demo operator status. +func RecordReadBillingDemoOperatorStatus(site, opClass, operatorKind, status, reason, modelVersion string) { + if ReadBillingDemoOperatorStatusCounter == nil || + site == "" || opClass == "" || operatorKind == "" || status == "" || reason == "" || modelVersion == "" { + return + } + ReadBillingDemoOperatorStatusCounter.WithLabelValues(site, opClass, operatorKind, status, reason, modelVersion).Inc() +} + +// AddReadBillingDemoBaseUnits records coefficient-free read billing demo base units. +func AddReadBillingDemoBaseUnits(site, opClass, operatorKind, unit, inputSource, inputSide, modelVersion string, value float64) { + if ReadBillingDemoBaseUnitsCounter == nil || + site == "" || opClass == "" || operatorKind == "" || unit == "" || inputSource == "" || inputSide == "" || modelVersion == "" || + value <= 0 { + return + } + ReadBillingDemoBaseUnitsCounter.WithLabelValues(site, opClass, operatorKind, unit, inputSource, inputSide, modelVersion).Add(value) +} + +// ObserveReadBillingDemoRowWidth records row-width factors used by read billing demo. +func ObserveReadBillingDemoRowWidth(site, opClass, operatorKind, rowWidthSource, modelVersion string, rowWidth float64) { + if ReadBillingDemoRowWidthHistogram == nil || + site == "" || opClass == "" || operatorKind == "" || rowWidthSource == "" || modelVersion == "" || rowWidth <= 0 { + return + } + ReadBillingDemoRowWidthHistogram.WithLabelValues(site, opClass, operatorKind, rowWidthSource, modelVersion).Observe(rowWidth) +} diff --git a/pkg/metrics/metrics.go b/pkg/metrics/metrics.go index a71cc5637d540..19031cc494802 100644 --- a/pkg/metrics/metrics.go +++ b/pkg/metrics/metrics.go @@ -102,6 +102,7 @@ func InitMetrics() { InitServerMetrics() InitSessionMetrics() InitRUV2Metrics() + InitExplainRUMetrics() InitSliMetrics() InitStatsMetrics() InitTelemetryMetrics() @@ -361,6 +362,17 @@ func RegisterMetrics() { prometheus.MustRegister(RUV2TiKVStorageProcessedKeysBatchGet) prometheus.MustRegister(RUV2TiKVStorageProcessedKeysGet) prometheus.MustRegister(RUV2TiKVCoprocessorWorkTotal) + prometheus.MustRegister(ExplainRUPreviewRUCounter) + prometheus.MustRegister(ExplainRUWorkRowsCounter) + prometheus.MustRegister(ExplainRUWorkBytesCounter) + prometheus.MustRegister(ExplainRURowWidthHistogram) + prometheus.MustRegister(ExplainRUStatementsCounter) + prometheus.MustRegister(ExplainRURenderDurationHistogram) + prometheus.MustRegister(ExplainRUComponentSnapshotCounter) + prometheus.MustRegister(ReadBillingDemoStatementsCounter) + prometheus.MustRegister(ReadBillingDemoOperatorStatusCounter) + prometheus.MustRegister(ReadBillingDemoBaseUnitsCounter) + prometheus.MustRegister(ReadBillingDemoRowWidthHistogram) prometheus.MustRegister(NetworkTransmissionStats) diff --git a/pkg/metrics/metrics_internal_test.go b/pkg/metrics/metrics_internal_test.go index a0fd8e1b66f3b..747f3e9eb05f5 100644 --- a/pkg/metrics/metrics_internal_test.go +++ b/pkg/metrics/metrics_internal_test.go @@ -95,6 +95,95 @@ func TestRUV2ExecutorCounterReturnsCachedKnownLabels(t *testing.T) { } } +func TestExplainRUMetrics(t *testing.T) { + InitExplainRUMetrics() + + RecordExplainRUStatus("success") + ObserveExplainRURenderDuration("success", 0.01) + RecordExplainRUComponentSnapshot("ok") + ObserveExplainRURow("plan", "", "projection", "read_billing_model", "plan_stats", 1.25, 3, 24, 8) + RecordReadBillingDemoStatement("success", "v1") + RecordReadBillingDemoOperatorStatus("tidb", "projection_eval", "projection", "ok", "none", "v1") + AddReadBillingDemoBaseUnits("tidb", "projection_eval", "projection", "input_rows", "runtime_act_rows", "all", "v1", 3) + ObserveReadBillingDemoRowWidth("tidb", "projection_eval", "projection", "plan_stats", "v1", 8) + + require.Equal(t, 1.0, readCounterValue(t, ExplainRUStatementsCounter.WithLabelValues("success"))) + require.Equal(t, 1.0, readCounterValue(t, ExplainRUComponentSnapshotCounter.WithLabelValues("ok"))) + require.Equal(t, 1.25, readCounterValue(t, ExplainRUPreviewRUCounter.WithLabelValues("plan", "", "projection", "read_billing_model"))) + require.Equal(t, 3.0, readCounterValue(t, ExplainRUWorkRowsCounter.WithLabelValues("plan", "", "projection", "read_billing_model"))) + require.Equal(t, 24.0, readCounterValue(t, ExplainRUWorkBytesCounter.WithLabelValues("plan", "", "projection", "read_billing_model"))) + require.Equal(t, 1.0, readCounterValue(t, ReadBillingDemoStatementsCounter.WithLabelValues("success", "v1"))) + require.Equal(t, 1.0, readCounterValue(t, ReadBillingDemoOperatorStatusCounter.WithLabelValues("tidb", "projection_eval", "projection", "ok", "none", "v1"))) + require.Equal(t, 3.0, readCounterValue(t, ReadBillingDemoBaseUnitsCounter.WithLabelValues("tidb", "projection_eval", "projection", "input_rows", "runtime_act_rows", "all", "v1"))) + + registry := prometheus.NewRegistry() + require.NoError(t, registry.Register(ExplainRUPreviewRUCounter)) + require.NoError(t, registry.Register(ExplainRUWorkRowsCounter)) + require.NoError(t, registry.Register(ExplainRUWorkBytesCounter)) + require.NoError(t, registry.Register(ExplainRURowWidthHistogram)) + require.NoError(t, registry.Register(ExplainRUStatementsCounter)) + require.NoError(t, registry.Register(ExplainRURenderDurationHistogram)) + require.NoError(t, registry.Register(ExplainRUComponentSnapshotCounter)) + require.NoError(t, registry.Register(ReadBillingDemoStatementsCounter)) + require.NoError(t, registry.Register(ReadBillingDemoOperatorStatusCounter)) + require.NoError(t, registry.Register(ReadBillingDemoBaseUnitsCounter)) + require.NoError(t, registry.Register(ReadBillingDemoRowWidthHistogram)) + families, err := registry.Gather() + require.NoError(t, err) + require.NotNil(t, findMetricFamily(families, "tidb_explain_ru_preview_ru_total")) + require.NotNil(t, findMetricFamily(families, "tidb_explain_ru_work_rows_total")) + require.NotNil(t, findMetricFamily(families, "tidb_explain_ru_work_bytes_total")) + rowWidthFamily := findMetricFamily(families, "tidb_explain_ru_row_width_bytes") + require.NotNil(t, rowWidthFamily) + requireMetricFamilyHasLabels(t, rowWidthFamily, "component", "operator", "source") + require.True(t, metricHasLabelValue(rowWidthFamily.GetMetric()[0], "source", "plan_stats")) + statementFamily := findMetricFamily(families, "tidb_explain_ru_statements_total") + require.NotNil(t, statementFamily) + requireMetricFamilyHasLabels(t, statementFamily, "status") + require.NotNil(t, findMetricFamily(families, "tidb_explain_ru_render_duration_seconds")) + componentSnapshotFamily := findMetricFamily(families, "tidb_explain_ru_component_snapshot_total") + require.NotNil(t, componentSnapshotFamily) + requireMetricFamilyHasLabels(t, componentSnapshotFamily, "component_snapshot_status") + readBillingStatementFamily := findMetricFamily(families, "tidb_read_billing_demo_statements_total") + require.NotNil(t, readBillingStatementFamily) + requireMetricFamilyHasLabels(t, readBillingStatementFamily, "status", "model_version") + readBillingOperatorFamily := findMetricFamily(families, "tidb_read_billing_demo_operator_status_total") + require.NotNil(t, readBillingOperatorFamily) + requireMetricFamilyHasLabels(t, readBillingOperatorFamily, "site", "op_class", "operator_kind", "status", "reason", "model_version") + readBillingBaseUnitFamily := findMetricFamily(families, "tidb_read_billing_demo_base_units_total") + require.NotNil(t, readBillingBaseUnitFamily) + requireMetricFamilyHasLabels(t, readBillingBaseUnitFamily, "site", "op_class", "operator_kind", "unit", "input_source", "input_side", "model_version") + readBillingRowWidthFamily := findMetricFamily(families, "tidb_read_billing_demo_row_width_bytes") + require.NotNil(t, readBillingRowWidthFamily) + requireMetricFamilyHasLabels(t, readBillingRowWidthFamily, "site", "op_class", "operator_kind", "row_width_source", "model_version") +} + +func TestExplainRUMetricsIgnoreEmptyLabelsAndMissingValues(t *testing.T) { + InitExplainRUMetrics() + + RecordExplainRUStatus("") + ObserveExplainRURenderDuration("", 0.01) + RecordExplainRUComponentSnapshot("") + ObserveExplainRURow("summary", "total_preview_ru", "", "summary_total", "", 0, -1, -1, -1) + ObserveExplainRURow("plan", "", "", "read_billing_model", "operator_helper", -1, -1, -1, 32) + RecordReadBillingDemoStatement("", "v1") + RecordReadBillingDemoOperatorStatus("tidb", "projection_eval", "projection", "", "none", "v1") + AddReadBillingDemoBaseUnits("tidb", "projection_eval", "projection", "input_rows", "runtime_act_rows", "all", "v1", 0) + ObserveReadBillingDemoRowWidth("tidb", "projection_eval", "projection", "plan_stats", "v1", 0) + + require.Equal(t, 0, countCollectedMetrics(ExplainRUStatementsCounter)) + require.Equal(t, 0, countCollectedMetrics(ExplainRURenderDurationHistogram)) + require.Equal(t, 0, countCollectedMetrics(ExplainRUComponentSnapshotCounter)) + require.Equal(t, 1, countCollectedMetrics(ExplainRUPreviewRUCounter)) + require.Equal(t, 0, countCollectedMetrics(ExplainRUWorkRowsCounter)) + require.Equal(t, 0, countCollectedMetrics(ExplainRUWorkBytesCounter)) + require.Equal(t, 0, countCollectedMetrics(ExplainRURowWidthHistogram)) + require.Equal(t, 0, countCollectedMetrics(ReadBillingDemoStatementsCounter)) + require.Equal(t, 0, countCollectedMetrics(ReadBillingDemoOperatorStatusCounter)) + require.Equal(t, 0, countCollectedMetrics(ReadBillingDemoBaseUnitsCounter)) + require.Equal(t, 0, countCollectedMetrics(ReadBillingDemoRowWidthHistogram)) +} + func TestStmtSummaryMetricLabels(t *testing.T) { InitStmtSummaryMetrics() require.Equal(t, 0, countCollectedMetrics(StmtSummaryWindowRecordCount)) @@ -218,3 +307,16 @@ func metricHasLabelValue(metric *dto.Metric, name string, value string) bool { } return false } + +func requireMetricFamilyHasLabels(t *testing.T, family *dto.MetricFamily, names ...string) { + t.Helper() + require.NotEmpty(t, family.GetMetric()) + labels := make(map[string]struct{}, len(family.GetMetric()[0].GetLabel())) + for _, label := range family.GetMetric()[0].GetLabel() { + labels[label.GetName()] = struct{}{} + } + for _, name := range names { + _, ok := labels[name] + require.Truef(t, ok, "metric %s missing label %s", family.GetName(), name) + } +} diff --git a/pkg/planner/core/BUILD.bazel b/pkg/planner/core/BUILD.bazel index 2bf15ab229081..e7f2e928a2ea9 100644 --- a/pkg/planner/core/BUILD.bazel +++ b/pkg/planner/core/BUILD.bazel @@ -14,6 +14,7 @@ go_library( "core_init.go", "encode.go", "exhaust_physical_plans.go", + "explain_ru.go", "expression_codec_fn.go", "expression_rewriter.go", "find_best_task.go", @@ -193,6 +194,7 @@ go_library( "//pkg/util/size", "//pkg/util/slice", "//pkg/util/sqlexec", + "//pkg/util/stmtsummary", "//pkg/util/stringutil", "//pkg/util/syncutil", "//pkg/util/texttree", @@ -207,6 +209,7 @@ go_library( "@com_github_pingcap_tipb//go-tipb", "@com_github_tikv_client_go_v2//kv", "@com_github_tikv_client_go_v2//oracle", + "@com_github_tikv_pd_client//resource_group/controller", "@org_uber_go_atomic//:atomic", "@org_uber_go_zap//:zap", ], @@ -321,6 +324,7 @@ go_test( "//pkg/util/context", "//pkg/util/dbterror", "//pkg/util/dbterror/plannererrors", + "//pkg/util/execdetails", "//pkg/util/hint", "//pkg/util/logutil", "//pkg/util/mock", @@ -332,6 +336,8 @@ go_test( "@com_github_pingcap_failpoint//:failpoint", "@com_github_pingcap_tipb//go-tipb", "@com_github_stretchr_testify//require", + "@com_github_tikv_client_go_v2//util", + "@com_github_tikv_pd_client//resource_group/controller", "@org_uber_go_goleak//:goleak", "@org_uber_go_zap//:zap", "@org_uber_go_zap//zaptest/observer", diff --git a/pkg/planner/core/common_plans.go b/pkg/planner/core/common_plans.go index ae124a35819e5..0fb06300b2094 100644 --- a/pkg/planner/core/common_plans.go +++ b/pkg/planner/core/common_plans.go @@ -651,8 +651,9 @@ type Explain struct { ExecStmt ast.StmtNode RuntimeStatsColl *execdetails.RuntimeStatsColl - Rows [][]string - BriefBinaryPlan string + Rows [][]string + BriefBinaryPlan string + ruStatusRecorded bool } // GetBriefBinaryPlan returns the binary plan of the plan for explainfor. @@ -735,6 +736,8 @@ func (e *Explain) prepareSchema() error { fieldNames = []string{"binary plan"} case format == types.ExplainFormatTiDBJSON: fieldNames = []string{"TiDB_JSON"} + case format == types.ExplainFormatRU && e.Analyze: + fieldNames = []string{"section", "id", "component", "operatorClass", "actRows", "inputRows", "outputRows", "rowWidth", "rowWidthSource", "workRows", "workBytes", "unit", "count", "weight", "previewRU", "source", "note"} case e.Explore: fieldNames = []string{"statement", "binding_hint", "plan", "plan_digest", "avg_latency", "exec_times", "avg_scan_rows", "avg_returned_rows", "latency_per_returned_row", "scan_rows_per_returned_row", "recommend", "reason", @@ -932,6 +935,11 @@ func (e *Explain) RenderResult() error { return err } e.Rows = append(e.Rows, []string{str}) + case types.ExplainFormatRU: + if err := e.renderRUExplain(); err != nil { + e.recordExplainRUStatus(explainRUStatusError) + return err + } default: return errors.Errorf("explain format '%s' is not supported now", e.Format) } diff --git a/pkg/planner/core/common_plans_test.go b/pkg/planner/core/common_plans_test.go index cec90eef77648..7a72238f8186a 100644 --- a/pkg/planner/core/common_plans_test.go +++ b/pkg/planner/core/common_plans_test.go @@ -16,10 +16,23 @@ package core import ( "testing" + "time" + "github.com/pingcap/tidb/pkg/expression" + "github.com/pingcap/tidb/pkg/kv" "github.com/pingcap/tidb/pkg/parser" "github.com/pingcap/tidb/pkg/parser/ast" + "github.com/pingcap/tidb/pkg/parser/mysql" + "github.com/pingcap/tidb/pkg/planner/core/operator/physicalop" + "github.com/pingcap/tidb/pkg/planner/property" + "github.com/pingcap/tidb/pkg/statistics" + "github.com/pingcap/tidb/pkg/types" + "github.com/pingcap/tidb/pkg/util/execdetails" + "github.com/pingcap/tidb/pkg/util/mock" + "github.com/pingcap/tipb/go-tipb" "github.com/stretchr/testify/require" + tikvutil "github.com/tikv/client-go/v2/util" + rmclient "github.com/tikv/pd/client/resource_group/controller" ) func TestNewLineFieldsInfo(t *testing.T) { @@ -114,3 +127,414 @@ func TestNewLineFieldsInfo(t *testing.T) { require.Equal(t, c.expected, lineFieldsInfo) } } + +func TestExplainRUSelectGateStatus(t *testing.T) { + cases := []struct { + sql string + expected explainRUStatus + }{ + {"explain analyze format='ru' select 1", explainRUStatusSuccess}, + {"explain analyze format='ru' with cte as (select 1) select * from cte", explainRUStatusSuccess}, + {"explain analyze format='ru' select rand(), uuid()", explainRUStatusSuccess}, + {"explain analyze format='ru' select last_insert_id()", explainRUStatusSuccess}, + {"explain analyze format='ru' insert into t values (1)", explainRUStatusUnsupportedNonSelect}, + {"explain analyze format='ru' table t", explainRUStatusUnsupportedNonSelect}, + {"explain analyze format='ru' select 1 union table t", explainRUStatusUnsupportedNonSelect}, + {"explain analyze format='ru' select 1 into outfile '/tmp/explain_ru.csv'", explainRUStatusUnsupportedSideEffecting}, + {"explain analyze format='ru' select @a := 1", explainRUStatusUnsupportedSideEffecting}, + {"explain analyze format='ru' select 1 union select @a := 2", explainRUStatusUnsupportedSideEffecting}, + {"explain analyze format='ru' with cte as (select get_lock('x', 0)) select * from cte", explainRUStatusUnsupportedSideEffecting}, + {"explain analyze format='ru' select release_lock('x')", explainRUStatusUnsupportedSideEffecting}, + {"explain analyze format='ru' select release_all_locks()", explainRUStatusUnsupportedSideEffecting}, + {"explain analyze format='ru' select last_insert_id(1)", explainRUStatusUnsupportedSideEffecting}, + {"explain analyze format='ru' select nextval(seq)", explainRUStatusUnsupportedSideEffecting}, + {"explain analyze format='ru' select setval(seq, 1)", explainRUStatusUnsupportedSideEffecting}, + {"explain analyze format='ru' select sleep(1)", explainRUStatusUnsupportedSideEffecting}, + {"explain analyze format='ru' select * from t for update skip locked", explainRUStatusUnsupportedLockingSelect}, + {"explain analyze format='ru' select * from t for share skip locked", explainRUStatusUnsupportedLockingSelect}, + {"explain analyze format='ru' select 1 union select * from t for update", explainRUStatusUnsupportedLockingSelect}, + } + p := parser.New() + for _, tc := range cases { + stmt, err := p.ParseOneStmt(tc.sql, "", "") + require.NoError(t, err, tc.sql) + explain := stmt.(*ast.ExplainStmt) + require.Equal(t, tc.expected, explainRUSelectGateStatus(explain.Stmt), tc.sql) + } + require.Equal(t, explainRUStatusUnsupportedNonSelect, explainRUSelectGateStatus(&ast.SelectStmt{Kind: ast.SelectStmtKindValues})) + require.Equal(t, explainRUStatusUnsupportedNonSelect, explainRUValidateSetOprSelectList(&ast.SetOprSelectList{ + Selects: []ast.Node{&ast.SelectStmt{Kind: ast.SelectStmtKindValues}}, + })) +} + +func TestExplainRURowFormatting(t *testing.T) { + row := explainRURow{ + section: explainRUSectionPlan, + id: "Projection_1", + component: "projection", + operatorClass: "tidb/projection_eval", + actRows: 1, + hasActRows: true, + inputRows: 2, + hasInputRows: true, + outputRows: 1, + hasOutputRows: true, + rowWidth: 8, + hasRowWidth: true, + rowWidthSource: explainRUWidthSourcePlanStats, + workRows: 2, + hasWorkRows: true, + unit: readBillingDemoUnitInputRows, + count: 2, + hasCount: true, + weight: 0.25, + hasWeight: true, + previewRU: 6, + hasPreviewRU: true, + source: readBillingDemoInputSourceRuntimeRows, + note: "input_side=all,weight_version=v1", + } + require.Equal(t, []string{ + "plan", "Projection_1", "projection", "tidb/projection_eval", "1", "2", "1", "8.000000", "plan_stats", "2", "", "input_rows", "2", "0.250000", "6.000000", "runtime_act_rows", "input_side=all,weight_version=v1", + }, row.toStrings()) +} + +func TestExplainRUPlanFormulaUsesRowsAndModeledBytes(t *testing.T) { + tidbWeights, ok := readBillingDemoResolveWeights(readBillingDemoSiteTiDB, readBillingDemoOpClassProjection, readBillingDemoWeightVersion) + require.True(t, ok) + tikvWeights, ok := readBillingDemoResolveWeights(readBillingDemoSiteTiKV, readBillingDemoOpClassProjection, readBillingDemoWeightVersion) + require.True(t, ok) + require.NotEqual(t, tidbWeights, tikvWeights) + _, ok = readBillingDemoResolveWeights(readBillingDemoSiteTiKV, readBillingDemoOpClassPointLookup, readBillingDemoWeightVersion) + require.True(t, ok) + _, ok = readBillingDemoResolveWeights(readBillingDemoSiteTiDB, readBillingDemoOpClassPointLookup, readBillingDemoWeightVersion) + require.False(t, ok) + + weight, previewRU, ok := readBillingDemoUnitPreviewRU( + readBillingDemoUnit{unit: readBillingDemoUnitInputBytes, value: 4096}, + tidbWeights, + ) + require.True(t, ok) + require.Equal(t, tidbWeights.byte, weight) + require.Equal(t, 4096*tidbWeights.byte, previewRU) + + _, _, ok = readBillingDemoUnitPreviewRU(readBillingDemoUnit{unit: "scan_total_keys", value: 4}, tidbWeights) + require.False(t, ok) + + ctx := mock.NewContext() + col := &expression.Column{RetType: types.NewFieldType(mysql.TypeLonglong)} + schema := expression.NewSchema(col) + stats := &property.StatsInfo{RowCount: 5} + for _, tc := range []struct { + name string + site string + opClass string + op func() *FlatOperator + }{ + { + name: "range scan", + site: readBillingDemoSiteTiKV, + opClass: readBillingDemoOpClassRangeScan, + op: func() *FlatOperator { + scan := physicalop.PhysicalIndexScan{}.Init(ctx, 0) + return &FlatOperator{Origin: scan, IsRoot: false, StoreType: kv.TiKV} + }, + }, + { + name: "filter", + site: readBillingDemoSiteTiKV, + opClass: readBillingDemoOpClassFilter, + op: func() *FlatOperator { + return &FlatOperator{Origin: physicalop.PhysicalSelection{}.Init(ctx, stats, 0), IsRoot: false, StoreType: kv.TiKV} + }, + }, + { + name: "projection", + site: readBillingDemoSiteTiKV, + opClass: readBillingDemoOpClassProjection, + op: func() *FlatOperator { + return &FlatOperator{Origin: physicalop.PhysicalProjection{}.Init(ctx, stats, 0), IsRoot: false, StoreType: kv.TiKV} + }, + }, + { + name: "limit", + site: readBillingDemoSiteTiKV, + opClass: readBillingDemoOpClassLimit, + op: func() *FlatOperator { + return &FlatOperator{Origin: physicalop.PhysicalLimit{}.Init(ctx, stats, 0), IsRoot: false, StoreType: kv.TiKV} + }, + }, + { + name: "topn", + site: readBillingDemoSiteTiKV, + opClass: readBillingDemoOpClassTopN, + op: func() *FlatOperator { + return &FlatOperator{Origin: physicalop.PhysicalTopN{}.Init(ctx, stats, 0), IsRoot: false, StoreType: kv.TiKV} + }, + }, + { + name: "hash agg", + site: readBillingDemoSiteTiKV, + opClass: readBillingDemoOpClassHashAgg, + op: func() *FlatOperator { + return &FlatOperator{Origin: (&physicalop.BasePhysicalAgg{}).InitForHash(ctx, stats, 0, schema), IsRoot: false, StoreType: kv.TiKV} + }, + }, + { + name: "stream agg", + site: readBillingDemoSiteTiKV, + opClass: readBillingDemoOpClassStreamAgg, + op: func() *FlatOperator { + return &FlatOperator{Origin: (&physicalop.BasePhysicalAgg{}).InitForStream(ctx, stats, 0, schema), IsRoot: false, StoreType: kv.TiKV} + }, + }, + } { + t.Run(tc.site+" "+tc.name, func(t *testing.T) { + requireReadBillingDemoClass(t, tc.op(), tc.site, tc.opClass, true, "") + }) + } + requireReadBillingDemoClass(t, &FlatOperator{ + Origin: physicalop.PhysicalIndexScan{}.Init(ctx, 0), + IsRoot: false, + StoreType: kv.TiFlash, + }, readBillingDemoSiteTiKV, readBillingDemoOpClassRangeScan, false, readBillingDemoReasonUnsupportedTiFlash) + requireReadBillingDemoClass(t, &FlatOperator{ + Origin: physicalop.PhysicalExchangeReceiver{}.Init(ctx, stats), + IsRoot: true, + }, readBillingDemoSiteTiDB, readBillingDemoOpClassReaderReceive, false, readBillingDemoReasonUnsupportedMPP) + indexMerge := &physicalop.PhysicalIndexMergeReader{} + indexMerge.BasePhysicalPlan = physicalop.NewBasePhysicalPlan(ctx, "IndexMerge", indexMerge, 0) + requireReadBillingDemoClass(t, &FlatOperator{ + Origin: indexMerge, + IsRoot: true, + }, readBillingDemoSiteTiDB, readBillingDemoOpClassLookupReader, false, readBillingDemoReasonUnsupportedIndexMerge) +} + +func requireReadBillingDemoClass(t *testing.T, op *FlatOperator, site, opClass string, supported bool, reason string) { + t.Helper() + operator, ok, actualReason := readBillingDemoClassifyOperator(op) + require.Equal(t, supported, ok) + require.Equal(t, reason, actualReason) + require.Equal(t, site, operator.site) + require.Equal(t, opClass, operator.opClass) + if supported && readBillingDemoOperatorBillable(operator) { + _, hasWeights := readBillingDemoResolveWeights(operator.site, operator.opClass, readBillingDemoWeightVersion) + require.True(t, hasWeights, "missing read billing demo weights for %s/%s", operator.site, operator.opClass) + } +} + +func TestExplainRUComponentSnapshotStatusAndWeights(t *testing.T) { + require.Equal(t, explainRUComponentSnapshotMissing, extractExplainRUTestSnapshotStatus(nil)) + require.Equal(t, explainRUComponentSnapshotMissing, extractExplainRUTestSnapshotStatus(&execdetails.RURuntimeStats{})) + require.Equal(t, explainRUComponentSnapshotNonV2, extractExplainRUTestSnapshotStatus(&execdetails.RURuntimeStats{ + RUVersion: rmclient.RUVersionV1, + Metrics: &execdetails.RUV2Metrics{}, + })) + require.Equal(t, explainRUComponentSnapshotNilMetrics, extractExplainRUTestSnapshotStatus(&execdetails.RURuntimeStats{ + RUVersion: rmclient.RUVersionV2, + })) + + bypassedMetrics := &execdetails.RUV2Metrics{} + bypassedMetrics.SetBypass(true) + require.Equal(t, explainRUComponentSnapshotBypassed, extractExplainRUTestSnapshotStatus(&execdetails.RURuntimeStats{ + RUVersion: rmclient.RUVersionV2, + Metrics: bypassedMetrics, + })) + + okStats := &execdetails.RURuntimeStats{ + RUVersion: rmclient.RUVersionV2, + Metrics: &execdetails.RUV2Metrics{}, + } + snapshot, status := extractExplainRUTestSnapshot(okStats) + require.Equal(t, explainRUComponentSnapshotOK, status) + require.Same(t, okStats, snapshot) +} + +func extractExplainRUTestSnapshotStatus(stats *execdetails.RURuntimeStats) explainRUComponentSnapshotStatus { + _, status := extractExplainRUTestSnapshot(stats) + return status +} + +func extractExplainRUTestSnapshot(stats *execdetails.RURuntimeStats) (*execdetails.RURuntimeStats, explainRUComponentSnapshotStatus) { + coll := execdetails.NewRuntimeStatsColl(nil) + if stats != nil && (stats.RUVersion != 0 || stats.Metrics != nil) { + coll.RegisterStats(1, stats) + } + return explainRUExtractComponentSnapshot(coll, 1) +} + +func TestReadBillingDemoDirectCopInputRowsAndWidthUsesChildRows(t *testing.T) { + ctx := mock.NewContext() + col := &expression.Column{RetType: types.NewFieldType(mysql.TypeLonglong)} + schema := expression.NewSchema(col) + stats := &property.StatsInfo{RowCount: 5} + parent := (&physicalop.PhysicalHashAgg{}).InitForHash(ctx, stats, 0, schema).(*physicalop.PhysicalHashAgg) + child := (&physicalop.PhysicalSelection{}).Init(ctx, stats, 0) + childInput := physicalop.PhysicalTableScan{}.Init(ctx, 0) + childInput.SetSchema(schema) + child.SetChildren(childInput) + tree := FlatPlanTree{ + {Origin: parent, ChildrenIdx: []int{1}, ChildrenEndIdx: 1, IsRoot: false, StoreType: kv.TiKV}, + {Origin: child, IsRoot: false, StoreType: kv.TiKV}, + } + + runtimeStats := execdetails.NewRuntimeStatsColl(nil) + recordCopRows := func(planID int, rows uint64) { + iterations := uint64(1) + duration := uint64(1) + runtimeStats.RecordOneCopTask(planID, kv.TiKV, &tipb.ExecutorExecutionSummary{ + NumProducedRows: &rows, + NumIterations: &iterations, + TimeProcessedNs: &duration, + }) + } + recordCopRows(parent.ID(), 1) + recordCopRows(child.ID(), 5) + + proj := physicalop.PhysicalProjection{}.Init(ctx, stats, 0) + proj.SetSchema(schema) + operator, supported, reason := readBillingDemoClassifyOperator(&FlatOperator{ + Origin: proj, + IsRoot: false, + StoreType: kv.TiKV, + }) + require.True(t, supported) + require.Empty(t, reason) + require.Equal(t, readBillingDemoSiteTiKV, operator.site) + require.Equal(t, readBillingDemoOpClassProjection, operator.opClass) + + rows, width, widthSource, ok := readBillingDemoDirectCopInputRowsAndWidth( + ctx, + runtimeStats, + tree, + 0, + 8, + explainRUWidthSourceSchemaFallback, + runtimeStats.GetCopStats(parent.ID()), + ) + require.True(t, ok) + require.Equal(t, int64(5), rows) + require.Equal(t, float64(8), width) + require.Equal(t, explainRUWidthSourceSchemaTypeWidth, widthSource) +} + +func TestReadBillingDemoRangeScanKeepsFixedEventForEmptyInput(t *testing.T) { + ctx := mock.NewContext() + col := &expression.Column{RetType: types.NewFieldType(mysql.TypeLonglong)} + schema := expression.NewSchema(col) + scan := physicalop.PhysicalTableScan{}.Init(ctx, 0) + scan.SetSchema(schema) + scan.StoreType = kv.TiKV + scan.TblColHists = &statistics.HistColl{Pseudo: true} + scan.TblCols = []*expression.Column{col} + tree := FlatPlanTree{ + {Origin: scan, IsRoot: false, StoreType: kv.TiKV}, + } + + buildUnits := func(scanDetail *tikvutil.ScanDetail, producedRows uint64) ([]readBillingDemoUnit, bool) { + runtimeStats := execdetails.NewRuntimeStatsColl(nil) + runtimeStats.RecordCopStats(scan.ID(), kv.TiKV, scanDetail, tikvutil.TimeDetail{}, nil) + iterations := uint64(1) + duration := uint64(1) + runtimeStats.RecordOneCopTask(scan.ID(), kv.TiKV, &tipb.ExecutorExecutionSummary{ + NumProducedRows: &producedRows, + NumIterations: &iterations, + TimeProcessedNs: &duration, + }) + return readBillingDemoCopUnits( + ctx, + runtimeStats, + tree, + 0, + tree[0], + readBillingDemoOperatorResult{site: readBillingDemoSiteTiKV, opClass: readBillingDemoOpClassRangeScan, operatorKind: "tablescan"}, + ) + } + + units, ok := buildUnits(&tikvutil.ScanDetail{}, 0) + require.True(t, ok) + require.Equal(t, 1.0, readBillingDemoUnitValue(units, readBillingDemoUnitFixedEvents, readBillingDemoInputSideAll)) + require.Equal(t, 0.0, readBillingDemoUnitValue(units, readBillingDemoUnitInputRows, readBillingDemoInputSideAll)) + require.Equal(t, 0.0, readBillingDemoUnitValue(units, readBillingDemoUnitInputBytes, readBillingDemoInputSideAll)) + require.Equal(t, readBillingDemoInputSourceScanDetail, readBillingDemoUnitSource(units, readBillingDemoUnitInputRows, readBillingDemoInputSideAll)) + require.False(t, readBillingDemoUnitExists(units, "scan_total_keys", readBillingDemoInputSideAll)) + require.False(t, readBillingDemoUnitExists(units, "processed_key_size", readBillingDemoInputSideAll)) + + units, ok = buildUnits(&tikvutil.ScanDetail{TotalKeys: 4, ProcessedKeysSize: 128}, 2) + require.True(t, ok) + require.Equal(t, 4.0, readBillingDemoUnitValue(units, readBillingDemoUnitInputRows, readBillingDemoInputSideAll)) + require.Equal(t, 128.0, readBillingDemoUnitValue(units, readBillingDemoUnitInputBytes, readBillingDemoInputSideAll)) + require.Equal(t, readBillingDemoInputSourceScanDetail, readBillingDemoUnitSource(units, readBillingDemoUnitInputRows, readBillingDemoInputSideAll)) + + units, ok = buildUnits(&tikvutil.ScanDetail{}, 2) + require.True(t, ok) + require.Equal(t, 2.0, readBillingDemoUnitValue(units, readBillingDemoUnitInputRows, readBillingDemoInputSideAll)) + require.Equal(t, readBillingDemoInputSourceRuntimeRows, readBillingDemoUnitSource(units, readBillingDemoUnitInputRows, readBillingDemoInputSideAll)) + require.Greater(t, readBillingDemoUnitValue(units, readBillingDemoUnitInputBytes, readBillingDemoInputSideAll), 0.0) +} + +func TestReadBillingDemoHashJoinUnitsUseBuildProbeSides(t *testing.T) { + ctx := mock.NewContext() + col := &expression.Column{RetType: types.NewFieldType(mysql.TypeLonglong)} + schema := expression.NewSchema(col) + stats := &property.StatsInfo{RowCount: 10} + join := (&physicalop.PhysicalHashJoin{}).Init(ctx, stats, 0) + left := physicalop.PhysicalProjection{}.Init(ctx, stats, 0) + right := physicalop.PhysicalProjection{}.Init(ctx, stats, 0) + join.SetSchema(schema) + left.SetSchema(schema) + right.SetSchema(schema) + tree := FlatPlanTree{ + {Origin: join, ChildrenIdx: []int{1, 2}, IsRoot: true}, + {Origin: left, IsRoot: true, Label: BuildSide}, + {Origin: right, IsRoot: true, Label: ProbeSide}, + } + + runtimeStats := execdetails.NewRuntimeStatsColl(nil) + recordRootRows := func(planID int, rows int) { + runtimeStats.GetBasicRuntimeStats(planID, true).Record(time.Millisecond, rows) + } + recordRootRows(join.ID(), 6) + recordRootRows(left.ID(), 4) + recordRootRows(right.ID(), 6) + + units, ok := readBillingDemoRootUnits( + ctx, + runtimeStats, + tree, + 0, + tree[0], + readBillingDemoOperatorResult{site: readBillingDemoSiteTiDB, opClass: readBillingDemoOpClassHashJoin, operatorKind: "hashjoin"}, + ) + require.True(t, ok) + require.Equal(t, 1.0, readBillingDemoUnitValue(units, readBillingDemoUnitFixedEvents, readBillingDemoInputSideAll)) + require.Equal(t, 4.0, readBillingDemoUnitValue(units, readBillingDemoUnitInputRows, readBillingDemoInputSideBuild)) + require.Equal(t, 6.0, readBillingDemoUnitValue(units, readBillingDemoUnitInputRows, readBillingDemoInputSideProbe)) +} + +func readBillingDemoUnitValue(units []readBillingDemoUnit, unitName, side string) float64 { + for _, unit := range units { + if unit.unit == unitName && unit.side == side { + return unit.value + } + } + return -1 +} + +func readBillingDemoUnitSource(units []readBillingDemoUnit, unitName, side string) string { + for _, unit := range units { + if unit.unit == unitName && unit.side == side { + return unit.source + } + } + return "" +} + +func readBillingDemoUnitExists(units []readBillingDemoUnit, unitName, side string) bool { + for _, unit := range units { + if unit.unit == unitName && unit.side == side { + return true + } + } + return false +} diff --git a/pkg/planner/core/explain_ru.go b/pkg/planner/core/explain_ru.go new file mode 100644 index 0000000000000..c77c8cbf7336d --- /dev/null +++ b/pkg/planner/core/explain_ru.go @@ -0,0 +1,1300 @@ +// Copyright 2026 PingCAP, Inc. +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package core + +import ( + "strconv" + "strings" + "time" + + "github.com/pingcap/errors" + "github.com/pingcap/tidb/pkg/kv" + "github.com/pingcap/tidb/pkg/metrics" + "github.com/pingcap/tidb/pkg/parser/ast" + "github.com/pingcap/tidb/pkg/planner/cardinality" + "github.com/pingcap/tidb/pkg/planner/core/base" + "github.com/pingcap/tidb/pkg/planner/core/operator/physicalop" + "github.com/pingcap/tidb/pkg/sessionctx" + "github.com/pingcap/tidb/pkg/types" + "github.com/pingcap/tidb/pkg/util/chunk" + "github.com/pingcap/tidb/pkg/util/execdetails" + "github.com/pingcap/tidb/pkg/util/plancodec" + "github.com/pingcap/tidb/pkg/util/stmtsummary" + rmclient "github.com/tikv/pd/client/resource_group/controller" +) + +type explainRUStatus string + +const ( + explainRUStatusSuccess explainRUStatus = "success" + explainRUStatusUnsupportedNonAnalyze explainRUStatus = "unsupported_non_analyze" + explainRUStatusUnsupportedNonSelect explainRUStatus = "unsupported_non_select" + explainRUStatusUnsupportedSideEffecting explainRUStatus = "unsupported_side_effecting_select" + explainRUStatusUnsupportedLockingSelect explainRUStatus = "unsupported_locking_select" + explainRUStatusUnsupportedForConnection explainRUStatus = "unsupported_for_connection" + explainRUStatusError explainRUStatus = "error" + explainRUComponentSnapshotOK explainRUComponentSnapshotStatus = "ok" + explainRUComponentSnapshotMissing explainRUComponentSnapshotStatus = "missing" + explainRUComponentSnapshotNonV2 explainRUComponentSnapshotStatus = "non_v2" + explainRUComponentSnapshotNilMetrics explainRUComponentSnapshotStatus = "nil_metrics" + explainRUComponentSnapshotBypassed explainRUComponentSnapshotStatus = "bypassed" + explainRUSectionSummary = "summary" + explainRUSectionPlan = "plan" + explainRUSourceSummaryTotal = "summary_total" + explainRUWidthSourceOperatorHelper = "operator_helper" + explainRUWidthSourcePlanStats = "plan_stats" + explainRUWidthSourceSchemaTypeWidth = "schema_type_width" + explainRUWidthSourceSchemaFallback = "schema_fallback" + + readBillingDemoModelVersion = "v1" + readBillingDemoWeightVersion = "v1" + readBillingDemoStatusSuccess = "success" + readBillingDemoStatusUnsupported = "unsupported" + readBillingDemoStatusUnknownInput = "unknown_input" + readBillingDemoStatusError = "error" + readBillingDemoStatusOperatorOK = "ok" + readBillingDemoReasonNone = "none" + readBillingDemoReasonStatementError = "statement_error" + readBillingDemoReasonMissingPlan = "missing_plan" + readBillingDemoReasonMissingRuntimeStats = "missing_runtime_stats" + readBillingDemoReasonMissingRuntimeRows = "missing_runtime_rows" + readBillingDemoReasonMissingScanDetail = "missing_scan_detail" + readBillingDemoReasonUnsupportedOperator = "unsupported_operator" + readBillingDemoReasonUnsupportedTiFlash = "unsupported_tiflash" + readBillingDemoReasonUnsupportedMPP = "unsupported_mpp" + readBillingDemoReasonUnsupportedIndexMerge = "unsupported_index_merge" + readBillingDemoReasonUnsupportedLock = "unsupported_lock" + readBillingDemoReasonNonBillable = "non_billable" + readBillingDemoSiteStatement = "statement" + readBillingDemoSiteTiDB = "tidb" + readBillingDemoSiteTiKV = "tikv" + readBillingDemoOpClassStatement = "statement" + readBillingDemoOpClassFilter = "filter_eval" + readBillingDemoOpClassProjection = "projection_eval" + readBillingDemoOpClassLimit = "row_limit" + readBillingDemoOpClassTopN = "bounded_topn" + readBillingDemoOpClassSort = "full_ordering" + readBillingDemoOpClassWindow = "window_eval" + readBillingDemoOpClassHashAgg = "agg_hash" + readBillingDemoOpClassStreamAgg = "agg_stream" + readBillingDemoOpClassHashJoin = "join_hash" + readBillingDemoOpClassMergeJoin = "join_merge" + readBillingDemoOpClassLookupJoin = "join_lookup" + readBillingDemoOpClassReaderReceive = "reader_receive" + readBillingDemoOpClassLookupReader = "lookup_reader" + readBillingDemoOpClassOverlayReader = "overlay_reader" + readBillingDemoOpClassMetadataReader = "metadata_reader" + readBillingDemoOpClassPointLookup = "kv_point_lookup" + readBillingDemoOpClassRangeScan = "kv_range_scan" + readBillingDemoOpClassWrapper = "wrapper" + readBillingDemoOpClassSynthetic = "synthetic_source" + readBillingDemoOperatorStatement = "statement" + readBillingDemoUnitFixedEvents = "fixed_events" + readBillingDemoUnitInputRows = "input_rows" + readBillingDemoUnitInputBytes = "input_bytes" + readBillingDemoInputSourceRuntimeRows = "runtime_act_rows" + readBillingDemoInputSourceScanDetail = "scan_detail" + readBillingDemoInputSideAll = "all" + readBillingDemoInputSideBuild = "build" + readBillingDemoInputSideProbe = "probe" + readBillingDemoInputSideLeft = "left" + readBillingDemoInputSideRight = "right" +) + +type explainRUComponentSnapshotStatus string + +type readBillingDemoUnit struct { + unit string + source string + side string + value float64 + rowWidth float64 + widthSource string +} + +type readBillingDemoOperatorResult struct { + id string + site string + opClass string + operatorKind string + status string + reason string + actRows int64 + hasActRows bool + units []readBillingDemoUnit +} + +type readBillingDemoResult struct { + status string + reason string + operators []readBillingDemoOperatorResult +} + +type readBillingDemoOperatorWeights struct { + fixedEvent float64 + row float64 + byte float64 +} + +type readBillingDemoWeightKey struct { + site string + opClass string + version string +} + +var readBillingDemoWeights = map[readBillingDemoWeightKey]readBillingDemoOperatorWeights{ + {readBillingDemoSiteTiKV, readBillingDemoOpClassRangeScan, readBillingDemoWeightVersion}: {fixedEvent: 0.070, row: 0.000045, byte: 0.000020}, + {readBillingDemoSiteTiKV, readBillingDemoOpClassFilter, readBillingDemoWeightVersion}: {fixedEvent: 0.020, row: 0.000040, byte: 0.000006}, + {readBillingDemoSiteTiKV, readBillingDemoOpClassProjection, readBillingDemoWeightVersion}: {fixedEvent: 0.020, row: 0.000030, byte: 0.000006}, + {readBillingDemoSiteTiKV, readBillingDemoOpClassLimit, readBillingDemoWeightVersion}: {fixedEvent: 0.010, row: 0.000008, byte: 0.000002}, + {readBillingDemoSiteTiKV, readBillingDemoOpClassTopN, readBillingDemoWeightVersion}: {fixedEvent: 0.060, row: 0.000075, byte: 0.000012}, + {readBillingDemoSiteTiKV, readBillingDemoOpClassHashAgg, readBillingDemoWeightVersion}: {fixedEvent: 0.080, row: 0.000100, byte: 0.000014}, + {readBillingDemoSiteTiKV, readBillingDemoOpClassStreamAgg, readBillingDemoWeightVersion}: {fixedEvent: 0.060, row: 0.000065, byte: 0.000010}, + {readBillingDemoSiteTiKV, readBillingDemoOpClassPointLookup, readBillingDemoWeightVersion}: {fixedEvent: 0.045, row: 0.000030, byte: 0.000012}, + {readBillingDemoSiteTiDB, readBillingDemoOpClassFilter, readBillingDemoWeightVersion}: {fixedEvent: 0.020, row: 0.000030, byte: 0.000005}, + {readBillingDemoSiteTiDB, readBillingDemoOpClassProjection, readBillingDemoWeightVersion}: {fixedEvent: 0.020, row: 0.000020, byte: 0.000004}, + {readBillingDemoSiteTiDB, readBillingDemoOpClassLimit, readBillingDemoWeightVersion}: {fixedEvent: 0.010, row: 0.000006, byte: 0.000001}, + {readBillingDemoSiteTiDB, readBillingDemoOpClassTopN, readBillingDemoWeightVersion}: {fixedEvent: 0.060, row: 0.000060, byte: 0.000010}, + {readBillingDemoSiteTiDB, readBillingDemoOpClassSort, readBillingDemoWeightVersion}: {fixedEvent: 0.080, row: 0.000070, byte: 0.000012}, + {readBillingDemoSiteTiDB, readBillingDemoOpClassWindow, readBillingDemoWeightVersion}: {fixedEvent: 0.070, row: 0.000070, byte: 0.000010}, + {readBillingDemoSiteTiDB, readBillingDemoOpClassHashAgg, readBillingDemoWeightVersion}: {fixedEvent: 0.070, row: 0.000085, byte: 0.000012}, + {readBillingDemoSiteTiDB, readBillingDemoOpClassStreamAgg, readBillingDemoWeightVersion}: {fixedEvent: 0.050, row: 0.000055, byte: 0.000008}, + {readBillingDemoSiteTiDB, readBillingDemoOpClassHashJoin, readBillingDemoWeightVersion}: {fixedEvent: 0.110, row: 0.000115, byte: 0.000020}, + {readBillingDemoSiteTiDB, readBillingDemoOpClassMergeJoin, readBillingDemoWeightVersion}: {fixedEvent: 0.090, row: 0.000075, byte: 0.000012}, + {readBillingDemoSiteTiDB, readBillingDemoOpClassLookupJoin, readBillingDemoWeightVersion}: {fixedEvent: 0.120, row: 0.000120, byte: 0.000020}, + {readBillingDemoSiteTiDB, readBillingDemoOpClassReaderReceive, readBillingDemoWeightVersion}: {fixedEvent: 0.040, row: 0.000025, byte: 0.000014}, + {readBillingDemoSiteTiDB, readBillingDemoOpClassLookupReader, readBillingDemoWeightVersion}: {fixedEvent: 0.070, row: 0.000045, byte: 0.000016}, + {readBillingDemoSiteTiDB, readBillingDemoOpClassOverlayReader, readBillingDemoWeightVersion}: {fixedEvent: 0.050, row: 0.000035, byte: 0.000012}, + {readBillingDemoSiteTiDB, readBillingDemoOpClassMetadataReader, readBillingDemoWeightVersion}: { + fixedEvent: 0.020, + row: 0.000008, + byte: 0.000002, + }, +} + +type explainRURow struct { + section string + id string + component string + operatorClass string + actRows int64 + hasActRows bool + inputRows int64 + hasInputRows bool + outputRows int64 + hasOutputRows bool + rowWidth float64 + hasRowWidth bool + rowWidthSource string + workRows int64 + hasWorkRows bool + workBytes float64 + hasWorkBytes bool + unit string + count int64 + hasCount bool + weight float64 + hasWeight bool + previewRU float64 + hasPreviewRU bool + source string + note string +} + +// RecordReadBillingDemoForStatement emits coefficient-free read billing demo +// metrics for a completed statement and returns the structured statement +// summary stats. It is intentionally independent from RU v2 billing/reporting +// and never calls resource-control reporters. +func RecordReadBillingDemoForStatement(sctx sessionctx.Context, plan base.Plan, stmt ast.StmtNode, execErr error) stmtsummary.ReadBillingDemoStatementStats { + if sctx == nil || sctx.GetSessionVars() == nil || !sctx.GetSessionVars().EnableReadBillingDemo { + return stmtsummary.ReadBillingDemoStatementStats{} + } + // Restricted/internal SQL is not external workload calibration input. + if sctx.GetSessionVars().InRestrictedSQL || sctx.GetSessionVars().StmtCtx.InRestrictedSQL { + return stmtsummary.ReadBillingDemoStatementStats{} + } + planCtx := readBillingDemoPlanContext(plan) + if planCtx == nil { + planCtx = sctx.GetPlanCtx() + } + result := buildReadBillingDemoResult(planCtx, plan, stmt, execErr) + recordReadBillingDemoResult(result) + return buildReadBillingDemoStatementStats(result) +} + +func buildReadBillingDemoResult(sctx base.PlanContext, plan base.Plan, stmt ast.StmtNode, execErr error) readBillingDemoResult { + if execErr != nil { + return readBillingDemoFailure(readBillingDemoStatusError, readBillingDemoReasonStatementError) + } + if plan == nil { + return readBillingDemoFailure(readBillingDemoStatusUnknownInput, readBillingDemoReasonMissingPlan) + } + if gateStatus := explainRUSelectGateStatus(stmt); gateStatus != explainRUStatusSuccess { + return readBillingDemoFailure(readBillingDemoStatusUnsupported, string(gateStatus)) + } + if sctx == nil || sctx.GetSessionVars() == nil || sctx.GetSessionVars().StmtCtx.RuntimeStatsColl == nil { + return readBillingDemoFailure(readBillingDemoStatusUnknownInput, readBillingDemoReasonMissingRuntimeStats) + } + flat := FlattenPhysicalPlan(plan, true) + if flat == nil || len(flat.Main) == 0 || flat.InExplain || flat.InExecute { + return readBillingDemoFailure(readBillingDemoStatusUnknownInput, readBillingDemoReasonMissingPlan) + } + + result := readBillingDemoResult{ + status: readBillingDemoStatusSuccess, + reason: readBillingDemoReasonNone, + } + planCtx := readBillingDemoPlanContext(plan) + runtimeStats := sctx.GetSessionVars().StmtCtx.RuntimeStatsColl + if status, op := appendReadBillingDemoTree(&result, planCtx, runtimeStats, flat.Main); status != readBillingDemoStatusSuccess { + return readBillingDemoFailedOperator(status, op) + } + for _, tree := range flat.CTEs { + if status, op := appendReadBillingDemoTree(&result, planCtx, runtimeStats, tree); status != readBillingDemoStatusSuccess { + return readBillingDemoFailedOperator(status, op) + } + } + for _, tree := range flat.ScalarSubQueries { + if status, op := appendReadBillingDemoTree(&result, planCtx, runtimeStats, tree); status != readBillingDemoStatusSuccess { + return readBillingDemoFailedOperator(status, op) + } + } + return result +} + +func readBillingDemoPlanContext(plan base.Plan) base.PlanContext { + if plan == nil { + return nil + } + return plan.SCtx() +} + +func readBillingDemoResolveWeights(site, opClass, version string) (readBillingDemoOperatorWeights, bool) { + weights, ok := readBillingDemoWeights[readBillingDemoWeightKey{site: site, opClass: opClass, version: version}] + return weights, ok +} + +func readBillingDemoUnitWeight(weights readBillingDemoOperatorWeights, unit string) (float64, bool) { + switch unit { + case readBillingDemoUnitFixedEvents: + return weights.fixedEvent, true + case readBillingDemoUnitInputRows: + return weights.row, true + case readBillingDemoUnitInputBytes: + return weights.byte, true + default: + return 0, false + } +} + +func readBillingDemoUnitPreviewRU(unit readBillingDemoUnit, weights readBillingDemoOperatorWeights) (float64, float64, bool) { + weight, ok := readBillingDemoUnitWeight(weights, unit.unit) + if !ok { + return 0, 0, false + } + return weight, unit.value * weight, true +} + +func readBillingDemoFailure(status, reason string) readBillingDemoResult { + return readBillingDemoResult{ + status: status, + reason: reason, + operators: []readBillingDemoOperatorResult{{ + site: readBillingDemoSiteStatement, + opClass: readBillingDemoOpClassStatement, + operatorKind: readBillingDemoOperatorStatement, + status: status, + reason: reason, + }}, + } +} + +func readBillingDemoFailedOperator(status string, op readBillingDemoOperatorResult) readBillingDemoResult { + op.status = status + if op.reason == "" { + op.reason = readBillingDemoReasonUnsupportedOperator + } + return readBillingDemoResult{ + status: status, + reason: op.reason, + operators: []readBillingDemoOperatorResult{op}, + } +} + +func summarizeReadBillingDemoBaseUnits(result readBillingDemoResult) stmtsummary.ReadBillingDemoBaseUnitSummary { + if result.status != readBillingDemoStatusSuccess { + return stmtsummary.ReadBillingDemoBaseUnitSummary{} + } + var summary stmtsummary.ReadBillingDemoBaseUnitSummary + for _, op := range result.operators { + if op.status != readBillingDemoStatusOperatorOK || !readBillingDemoOperatorBillable(op) { + continue + } + for _, unit := range op.units { + switch unit.unit { + case readBillingDemoUnitFixedEvents: + summary.SumReadBillingDemoFixedEvents += unit.value + case readBillingDemoUnitInputRows: + summary.SumReadBillingDemoInputRows += unit.value + case readBillingDemoUnitInputBytes: + summary.SumReadBillingDemoInputBytes += unit.value + } + } + } + return summary +} + +func buildReadBillingDemoStatementStats(result readBillingDemoResult) stmtsummary.ReadBillingDemoStatementStats { + stats := stmtsummary.ReadBillingDemoStatementStats{ + ModelVersion: readBillingDemoModelVersion, + WeightVersion: readBillingDemoWeightVersion, + } + status := result.status + if status == "" { + status = readBillingDemoStatusUnknownInput + } + reason := result.reason + if reason == "" { + reason = readBillingDemoReasonNone + } + stats.Statuses = append(stats.Statuses, stmtsummary.ReadBillingDemoStatusSample{ + ModelVersion: readBillingDemoModelVersion, + WeightVersion: readBillingDemoWeightVersion, + Site: readBillingDemoSiteStatement, + OpClass: readBillingDemoOpClassStatement, + OperatorKind: readBillingDemoOperatorStatement, + Status: status, + Reason: reason, + }) + for _, op := range result.operators { + opStatus := op.status + if opStatus == "" { + opStatus = status + } + opReason := op.reason + if opReason == "" { + opReason = reason + } + if opReason == "" { + opReason = readBillingDemoReasonNone + } + if !(op.site == readBillingDemoSiteStatement && + op.opClass == readBillingDemoOpClassStatement && + op.operatorKind == readBillingDemoOperatorStatement && + opStatus == status && + opReason == reason) { + stats.Statuses = append(stats.Statuses, stmtsummary.ReadBillingDemoStatusSample{ + ModelVersion: readBillingDemoModelVersion, + WeightVersion: readBillingDemoWeightVersion, + Site: op.site, + OpClass: op.opClass, + OperatorKind: op.operatorKind, + Status: opStatus, + Reason: opReason, + }) + } + if status != readBillingDemoStatusSuccess || opStatus != readBillingDemoStatusOperatorOK || !readBillingDemoOperatorBillable(op) { + continue + } + for _, unit := range op.units { + sample := stmtsummary.ReadBillingDemoBaseUnitSample{ + ModelVersion: readBillingDemoModelVersion, + WeightVersion: readBillingDemoWeightVersion, + Site: op.site, + OpClass: op.opClass, + OperatorKind: op.operatorKind, + Unit: unit.unit, + InputSource: unit.source, + InputSide: unit.side, + RowWidthSource: unit.widthSource, + Value: unit.value, + RowWidth: unit.rowWidth, + } + stats.BaseUnits = append(stats.BaseUnits, sample) + switch unit.unit { + case readBillingDemoUnitFixedEvents: + stats.Totals.SumReadBillingDemoFixedEvents += unit.value + case readBillingDemoUnitInputRows: + stats.Totals.SumReadBillingDemoInputRows += unit.value + case readBillingDemoUnitInputBytes: + stats.Totals.SumReadBillingDemoInputBytes += unit.value + } + } + } + return stats +} + +func appendReadBillingDemoTree(result *readBillingDemoResult, sctx base.PlanContext, runtimeStats *execdetails.RuntimeStatsColl, tree FlatPlanTree) (string, readBillingDemoOperatorResult) { + for i, op := range tree { + if op == nil || op.Origin == nil || op.Origin.ExplainID().String() == "_0" { + continue + } + operator, supported, reason := readBillingDemoClassifyOperator(op) + if !supported { + return readBillingDemoStatusUnsupported, operator.withReason(reason) + } + operator.id = op.ExplainID().String() + if actRows, ok := readBillingDemoOperatorActRows(runtimeStats, tree, i, op, operator); ok { + operator.actRows = actRows + operator.hasActRows = true + } + if !readBillingDemoOperatorBillable(operator) { + operator.status = readBillingDemoStatusOperatorOK + result.operators = append(result.operators, operator.withReason(readBillingDemoReasonNonBillable)) + continue + } + var units []readBillingDemoUnit + var ok bool + if op.IsRoot { + units, ok = readBillingDemoRootUnits(sctx, runtimeStats, tree, i, op, operator) + } else { + units, ok = readBillingDemoCopUnits(sctx, runtimeStats, tree, i, op, operator) + } + if !ok { + if op.IsRoot { + return readBillingDemoStatusUnknownInput, operator.withReason(readBillingDemoReasonMissingRuntimeRows) + } + return readBillingDemoStatusUnknownInput, operator.withReason(readBillingDemoReasonMissingScanDetail) + } + operator.status = readBillingDemoStatusOperatorOK + operator.reason = readBillingDemoReasonNone + operator.units = units + result.operators = append(result.operators, operator) + } + return readBillingDemoStatusSuccess, readBillingDemoOperatorResult{} +} + +func (op readBillingDemoOperatorResult) withReason(reason string) readBillingDemoOperatorResult { + op.reason = reason + return op +} + +func readBillingDemoOperatorBillable(op readBillingDemoOperatorResult) bool { + return op.opClass != readBillingDemoOpClassWrapper && op.opClass != readBillingDemoOpClassSynthetic +} + +func readBillingDemoOperatorActRows(runtimeStats *execdetails.RuntimeStatsColl, tree FlatPlanTree, idx int, op *FlatOperator, operator readBillingDemoOperatorResult) (int64, bool) { + if op == nil { + return 0, false + } + if op.IsRoot { + return readBillingDemoPlanActRows(runtimeStats, op.Origin.ID()) + } + if copStats := readBillingDemoCopStats(runtimeStats, tree, idx, operator.opClass); copStats != nil { + return copStats.GetActRows(), true + } + return 0, false +} + +func readBillingDemoClassifyOperator(op *FlatOperator) (readBillingDemoOperatorResult, bool, string) { + operatorKind := strings.ToLower(op.Origin.TP()) + if !op.IsRoot { + if op.StoreType == kv.TiFlash { + return readBillingDemoOperatorResult{site: readBillingDemoSiteTiKV, opClass: readBillingDemoOpClassRangeScan, operatorKind: operatorKind}, false, readBillingDemoReasonUnsupportedTiFlash + } + switch op.Origin.TP() { + case plancodec.TypeTableScan, plancodec.TypeIdxScan, + plancodec.TypeTableFullScan, plancodec.TypeTableRangeScan, plancodec.TypeTableRowIDScan, + plancodec.TypeIndexFullScan, plancodec.TypeIndexRangeScan: + return readBillingDemoOperatorResult{site: readBillingDemoSiteTiKV, opClass: readBillingDemoOpClassRangeScan, operatorKind: operatorKind}, true, "" + case plancodec.TypeSel: + return readBillingDemoOperatorResult{site: readBillingDemoSiteTiKV, opClass: readBillingDemoOpClassFilter, operatorKind: operatorKind}, true, "" + case plancodec.TypeProj: + return readBillingDemoOperatorResult{site: readBillingDemoSiteTiKV, opClass: readBillingDemoOpClassProjection, operatorKind: operatorKind}, true, "" + case plancodec.TypeLimit: + return readBillingDemoOperatorResult{site: readBillingDemoSiteTiKV, opClass: readBillingDemoOpClassLimit, operatorKind: operatorKind}, true, "" + case plancodec.TypeTopN: + return readBillingDemoOperatorResult{site: readBillingDemoSiteTiKV, opClass: readBillingDemoOpClassTopN, operatorKind: operatorKind}, true, "" + case plancodec.TypeHashAgg: + return readBillingDemoOperatorResult{site: readBillingDemoSiteTiKV, opClass: readBillingDemoOpClassHashAgg, operatorKind: operatorKind}, true, "" + case plancodec.TypeStreamAgg: + return readBillingDemoOperatorResult{site: readBillingDemoSiteTiKV, opClass: readBillingDemoOpClassStreamAgg, operatorKind: operatorKind}, true, "" + case plancodec.TypeSort, plancodec.TypeHashJoin, plancodec.TypeMergeJoin, plancodec.TypeIndexJoin: + return readBillingDemoOperatorResult{site: readBillingDemoSiteTiKV, opClass: readBillingDemoOpClassRangeScan, operatorKind: operatorKind}, false, readBillingDemoReasonUnsupportedOperator + default: + return readBillingDemoOperatorResult{site: readBillingDemoSiteTiKV, opClass: readBillingDemoOpClassRangeScan, operatorKind: operatorKind}, false, readBillingDemoReasonUnsupportedOperator + } + } + + switch op.Origin.TP() { + case plancodec.TypeExchangeReceiver, plancodec.TypeExchangeSender: + return readBillingDemoOperatorResult{site: readBillingDemoSiteTiDB, opClass: readBillingDemoOpClassReaderReceive, operatorKind: operatorKind}, false, readBillingDemoReasonUnsupportedMPP + case plancodec.TypeIndexMerge: + return readBillingDemoOperatorResult{site: readBillingDemoSiteTiDB, opClass: readBillingDemoOpClassLookupReader, operatorKind: operatorKind}, false, readBillingDemoReasonUnsupportedIndexMerge + case plancodec.TypeLock: + return readBillingDemoOperatorResult{site: readBillingDemoSiteTiDB, opClass: readBillingDemoOpClassReaderReceive, operatorKind: operatorKind}, false, readBillingDemoReasonUnsupportedLock + case plancodec.TypePointGet, plancodec.TypeBatchPointGet: + return readBillingDemoOperatorResult{site: readBillingDemoSiteTiKV, opClass: readBillingDemoOpClassPointLookup, operatorKind: operatorKind}, true, "" + case plancodec.TypeSel: + return readBillingDemoOperatorResult{site: readBillingDemoSiteTiDB, opClass: readBillingDemoOpClassFilter, operatorKind: operatorKind}, true, "" + case plancodec.TypeProj: + return readBillingDemoOperatorResult{site: readBillingDemoSiteTiDB, opClass: readBillingDemoOpClassProjection, operatorKind: operatorKind}, true, "" + case plancodec.TypeLimit: + return readBillingDemoOperatorResult{site: readBillingDemoSiteTiDB, opClass: readBillingDemoOpClassLimit, operatorKind: operatorKind}, true, "" + case plancodec.TypeTopN: + return readBillingDemoOperatorResult{site: readBillingDemoSiteTiDB, opClass: readBillingDemoOpClassTopN, operatorKind: operatorKind}, true, "" + case plancodec.TypeSort: + return readBillingDemoOperatorResult{site: readBillingDemoSiteTiDB, opClass: readBillingDemoOpClassSort, operatorKind: operatorKind}, true, "" + case plancodec.TypeWindow: + return readBillingDemoOperatorResult{site: readBillingDemoSiteTiDB, opClass: readBillingDemoOpClassWindow, operatorKind: operatorKind}, true, "" + case plancodec.TypeHashAgg: + return readBillingDemoOperatorResult{site: readBillingDemoSiteTiDB, opClass: readBillingDemoOpClassHashAgg, operatorKind: operatorKind}, true, "" + case plancodec.TypeStreamAgg: + return readBillingDemoOperatorResult{site: readBillingDemoSiteTiDB, opClass: readBillingDemoOpClassStreamAgg, operatorKind: operatorKind}, true, "" + case plancodec.TypeHashJoin: + return readBillingDemoOperatorResult{site: readBillingDemoSiteTiDB, opClass: readBillingDemoOpClassHashJoin, operatorKind: operatorKind}, true, "" + case plancodec.TypeMergeJoin: + return readBillingDemoOperatorResult{site: readBillingDemoSiteTiDB, opClass: readBillingDemoOpClassMergeJoin, operatorKind: operatorKind}, true, "" + case plancodec.TypeIndexJoin, plancodec.TypeIndexHashJoin, plancodec.TypeIndexMergeJoin: + return readBillingDemoOperatorResult{site: readBillingDemoSiteTiDB, opClass: readBillingDemoOpClassLookupJoin, operatorKind: operatorKind}, true, "" + case plancodec.TypeTableReader, plancodec.TypeIndexReader: + return readBillingDemoOperatorResult{site: readBillingDemoSiteTiDB, opClass: readBillingDemoOpClassReaderReceive, operatorKind: operatorKind}, true, "" + case plancodec.TypeIndexLookUp, plancodec.TypeLocalIndexLookUp: + return readBillingDemoOperatorResult{site: readBillingDemoSiteTiDB, opClass: readBillingDemoOpClassLookupReader, operatorKind: operatorKind}, true, "" + case plancodec.TypeUnionScan: + return readBillingDemoOperatorResult{site: readBillingDemoSiteTiDB, opClass: readBillingDemoOpClassOverlayReader, operatorKind: operatorKind}, true, "" + case plancodec.TypeMemTableScan, plancodec.TypeClusterMemTableReader: + return readBillingDemoOperatorResult{site: readBillingDemoSiteTiDB, opClass: readBillingDemoOpClassMetadataReader, operatorKind: operatorKind}, true, "" + case plancodec.TypeUnion: + return readBillingDemoOperatorResult{site: readBillingDemoSiteTiDB, opClass: readBillingDemoOpClassWrapper, operatorKind: operatorKind}, true, "" + case plancodec.TypeDual: + return readBillingDemoOperatorResult{site: readBillingDemoSiteTiDB, opClass: readBillingDemoOpClassSynthetic, operatorKind: operatorKind}, true, "" + default: + return readBillingDemoOperatorResult{site: readBillingDemoSiteTiDB, opClass: readBillingDemoOpClassReaderReceive, operatorKind: operatorKind}, false, readBillingDemoReasonUnsupportedOperator + } +} + +func readBillingDemoRootUnits(sctx base.PlanContext, runtimeStats *execdetails.RuntimeStatsColl, tree FlatPlanTree, idx int, op *FlatOperator, operator readBillingDemoOperatorResult) ([]readBillingDemoUnit, bool) { + outputRows, ok := readBillingDemoPlanActRows(runtimeStats, op.Origin.ID()) + if !ok { + return nil, false + } + rowWidth, rowWidthSource := explainRURowWidth(sctx, op.Origin, true) + units := []readBillingDemoUnit{{ + unit: readBillingDemoUnitFixedEvents, + source: readBillingDemoInputSourceRuntimeRows, + side: readBillingDemoInputSideAll, + value: 1, + rowWidth: rowWidth, + widthSource: rowWidthSource, + }} + switch operator.opClass { + case readBillingDemoOpClassHashJoin: + return appendReadBillingDemoJoinUnits(units, sctx, runtimeStats, tree, idx, rowWidth, rowWidthSource, true) + case readBillingDemoOpClassMergeJoin, readBillingDemoOpClassLookupJoin: + return appendReadBillingDemoJoinUnits(units, sctx, runtimeStats, tree, idx, rowWidth, rowWidthSource, false) + default: + inputRows, inputRowWidth, ok := readBillingDemoDirectLocalInputRowsAndWidth(sctx, runtimeStats, tree, idx, rowWidth) + if !ok { + return nil, false + } + if inputRows == 0 && (len(tree[idx].ChildrenIdx) == 0 || readBillingDemoUseOutputRowsAsInput(operator.opClass)) { + inputRows = outputRows + inputRowWidth = rowWidth + } + inputBytes := float64(inputRows) * inputRowWidth + units = append(units, + readBillingDemoUnit{unit: readBillingDemoUnitInputRows, source: readBillingDemoInputSourceRuntimeRows, side: readBillingDemoInputSideAll, value: float64(inputRows), rowWidth: inputRowWidth, widthSource: rowWidthSource}, + readBillingDemoUnit{unit: readBillingDemoUnitInputBytes, source: readBillingDemoInputSourceRuntimeRows, side: readBillingDemoInputSideAll, value: inputBytes, rowWidth: inputRowWidth, widthSource: rowWidthSource}, + ) + return units, true + } +} + +func readBillingDemoUseOutputRowsAsInput(opClass string) bool { + switch opClass { + case readBillingDemoOpClassReaderReceive, readBillingDemoOpClassLookupReader, readBillingDemoOpClassMetadataReader, readBillingDemoOpClassPointLookup: + return true + default: + return false + } +} + +func appendReadBillingDemoJoinUnits(units []readBillingDemoUnit, sctx base.PlanContext, runtimeStats *execdetails.RuntimeStatsColl, tree FlatPlanTree, idx int, fallbackWidth float64, fallbackWidthSource string, useBuildProbe bool) ([]readBillingDemoUnit, bool) { + for childOrder, childIdx := range tree[idx].ChildrenIdx { + if childIdx < 0 || childIdx >= len(tree) || tree[childIdx] == nil || !tree[childIdx].IsRoot { + continue + } + rows, ok := readBillingDemoPlanActRows(runtimeStats, tree[childIdx].Origin.ID()) + if !ok { + return nil, false + } + width, widthSource := explainRURowWidth(sctx, tree[childIdx].Origin, true) + if width <= 0 { + width = fallbackWidth + widthSource = fallbackWidthSource + } + side := readBillingDemoInputSideAll + if useBuildProbe { + switch tree[childIdx].Label { + case BuildSide: + side = readBillingDemoInputSideBuild + case ProbeSide: + side = readBillingDemoInputSideProbe + default: + if childOrder == 0 { + side = readBillingDemoInputSideBuild + } else { + side = readBillingDemoInputSideProbe + } + } + } else if childOrder == 0 { + side = readBillingDemoInputSideLeft + } else { + side = readBillingDemoInputSideRight + } + units = append(units, + readBillingDemoUnit{unit: readBillingDemoUnitInputRows, source: readBillingDemoInputSourceRuntimeRows, side: side, value: float64(rows), rowWidth: width, widthSource: widthSource}, + readBillingDemoUnit{unit: readBillingDemoUnitInputBytes, source: readBillingDemoInputSourceRuntimeRows, side: side, value: float64(rows) * width, rowWidth: width, widthSource: widthSource}, + ) + } + return units, true +} + +func readBillingDemoDirectLocalInputRowsAndWidth(sctx base.PlanContext, runtimeStats *execdetails.RuntimeStatsColl, tree FlatPlanTree, idx int, fallbackWidth float64) (int64, float64, bool) { + if idx < 0 || idx >= len(tree) || tree[idx] == nil { + return 0, fallbackWidth, true + } + var rows int64 + inputBytes := 0.0 + for _, childIdx := range tree[idx].ChildrenIdx { + if childIdx < 0 || childIdx >= len(tree) || tree[childIdx] == nil || !tree[childIdx].IsRoot { + continue + } + childRows, ok := readBillingDemoPlanActRows(runtimeStats, tree[childIdx].Origin.ID()) + if !ok { + return 0, 0, false + } + childWidth, _ := explainRURowWidth(sctx, tree[childIdx].Origin, true) + rows += childRows + inputBytes += float64(childRows) * childWidth + } + if rows == 0 { + return 0, fallbackWidth, true + } + return rows, inputBytes / float64(rows), true +} + +func readBillingDemoPlanActRows(runtimeStats *execdetails.RuntimeStatsColl, planID int) (int64, bool) { + if runtimeStats == nil || !runtimeStats.ExistsRootStats(planID) { + return 0, false + } + return runtimeStats.GetPlanActRows(planID), true +} + +func readBillingDemoCopUnits(sctx base.PlanContext, runtimeStats *execdetails.RuntimeStatsColl, tree FlatPlanTree, idx int, op *FlatOperator, operator readBillingDemoOperatorResult) ([]readBillingDemoUnit, bool) { + copStats := readBillingDemoCopStats(runtimeStats, tree, idx, operator.opClass) + if copStats == nil { + return nil, false + } + scanDetail := copStats.GetScanDetail() + rowWidth, rowWidthSource := explainRURowWidth(sctx, op.Origin, false) + units := []readBillingDemoUnit{{ + unit: readBillingDemoUnitFixedEvents, + source: readBillingDemoInputSourceScanDetail, + side: readBillingDemoInputSideAll, + value: 1, + rowWidth: rowWidth, + widthSource: rowWidthSource, + }} + switch operator.opClass { + case readBillingDemoOpClassRangeScan: + scanInputRows, scanInputBytes, scanInputSource := readBillingDemoRangeScanInput(scanDetail.TotalKeys, scanDetail.ProcessedKeysSize, copStats.GetActRows(), rowWidth) + units = append(units, + readBillingDemoUnit{unit: readBillingDemoUnitInputRows, source: scanInputSource, side: readBillingDemoInputSideAll, value: float64(scanInputRows), rowWidth: rowWidth, widthSource: rowWidthSource}, + readBillingDemoUnit{unit: readBillingDemoUnitInputBytes, source: scanInputSource, side: readBillingDemoInputSideAll, value: scanInputBytes, rowWidth: rowWidth, widthSource: rowWidthSource}, + ) + default: + rows, inputRowWidth, inputRowWidthSource, ok := readBillingDemoDirectCopInputRowsAndWidth(sctx, runtimeStats, tree, idx, rowWidth, rowWidthSource, copStats) + if !ok { + return nil, false + } + units = append(units, + readBillingDemoUnit{unit: readBillingDemoUnitInputRows, source: readBillingDemoInputSourceRuntimeRows, side: readBillingDemoInputSideAll, value: float64(rows), rowWidth: inputRowWidth, widthSource: inputRowWidthSource}, + readBillingDemoUnit{unit: readBillingDemoUnitInputBytes, source: readBillingDemoInputSourceRuntimeRows, side: readBillingDemoInputSideAll, value: float64(rows) * inputRowWidth, rowWidth: inputRowWidth, widthSource: inputRowWidthSource}, + ) + } + return units, true +} + +func readBillingDemoRangeScanInput(totalKeys, processedKeysSize, actRows int64, rowWidth float64) (int64, float64, string) { + if totalKeys != 0 || processedKeysSize != 0 { + inputBytes := float64(processedKeysSize) + if inputBytes == 0 && totalKeys > 0 { + inputBytes = float64(totalKeys) * rowWidth + } + return totalKeys, inputBytes, readBillingDemoInputSourceScanDetail + } + // Mock-store based tests can return executor rows without legacy scan-detail + // key counters. Keep the source explicit instead of pretending actRows are + // scan-detail TotalKeys. + if actRows > 0 { + return actRows, float64(actRows) * rowWidth, readBillingDemoInputSourceRuntimeRows + } + return 0, 0, readBillingDemoInputSourceScanDetail +} + +func readBillingDemoDirectCopInputRowsAndWidth( + sctx base.PlanContext, + runtimeStats *execdetails.RuntimeStatsColl, + tree FlatPlanTree, + idx int, + fallbackWidth float64, + fallbackWidthSource string, + ownStats *execdetails.CopRuntimeStats, +) (int64, float64, string, bool) { + var rows int64 + inputBytes := 0.0 + inputRowWidth := fallbackWidth + inputRowWidthSource := fallbackWidthSource + hasCopChild := false + for _, childIdx := range tree[idx].ChildrenIdx { + if childIdx < 0 || childIdx >= len(tree) || tree[childIdx] == nil || tree[childIdx].IsRoot { + continue + } + hasCopChild = true + childStats := runtimeStats.GetCopStats(tree[childIdx].Origin.ID()) + if childStats == nil { + return 0, 0, "", false + } + childRows := childStats.GetActRows() + childWidth, childWidthSource := explainRURowWidth(sctx, tree[childIdx].Origin, false) + if childWidth <= 0 { + childWidth = fallbackWidth + childWidthSource = fallbackWidthSource + } + inputRowWidth = childWidth + inputRowWidthSource = childWidthSource + rows += childRows + inputBytes += float64(childRows) * childWidth + } + if !hasCopChild { + return ownStats.GetActRows(), fallbackWidth, fallbackWidthSource, true + } + if rows == 0 { + return 0, inputRowWidth, inputRowWidthSource, true + } + return rows, inputBytes / float64(rows), inputRowWidthSource, true +} + +func readBillingDemoCopStats(runtimeStats *execdetails.RuntimeStatsColl, tree FlatPlanTree, idx int, opClass string) *execdetails.CopRuntimeStats { + if runtimeStats == nil || idx < 0 || idx >= len(tree) || tree[idx] == nil || tree[idx].Origin == nil { + return nil + } + exact := runtimeStats.GetCopStats(tree[idx].Origin.ID()) + if readBillingDemoCopStatsUsable(exact, opClass) { + return exact + } + // distsql may attach scan detail to the last cop plan ID in a task, not + // necessarily to the scan node. Search the same cop subtree before failing. + if end := tree[idx].ChildrenEndIdx; end > idx { + if end >= len(tree) { + end = len(tree) - 1 + } + for i := end; i > idx; i-- { + if tree[i] == nil || tree[i].IsRoot || tree[i].Origin == nil || tree[i].StoreType != tree[idx].StoreType { + continue + } + if stats := runtimeStats.GetCopStats(tree[i].Origin.ID()); readBillingDemoCopStatsUsable(stats, opClass) { + return stats + } + } + } + for i := idx - 1; i >= 0; i-- { + if tree[i] == nil || tree[i].IsRoot || tree[i].Origin == nil || tree[i].StoreType != tree[idx].StoreType || tree[i].ChildrenEndIdx < idx { + continue + } + if stats := runtimeStats.GetCopStats(tree[i].Origin.ID()); readBillingDemoCopStatsUsable(stats, opClass) { + return stats + } + } + return exact +} + +func readBillingDemoCopStatsUsable(copStats *execdetails.CopRuntimeStats, opClass string) bool { + if copStats == nil { + return false + } + if opClass != readBillingDemoOpClassRangeScan { + return true + } + scanDetail := copStats.GetScanDetail() + return scanDetail.TotalKeys != 0 || scanDetail.ProcessedKeysSize != 0 +} + +func recordReadBillingDemoResult(result readBillingDemoResult) { + status := result.status + if status == "" { + status = readBillingDemoStatusUnknownInput + } + metrics.RecordReadBillingDemoStatement(status, readBillingDemoModelVersion) + for _, op := range result.operators { + opStatus := op.status + if opStatus == "" { + opStatus = status + } + reason := op.reason + if reason == "" { + reason = result.reason + } + if reason == "" { + reason = readBillingDemoReasonNone + } + metrics.RecordReadBillingDemoOperatorStatus(op.site, op.opClass, op.operatorKind, opStatus, reason, readBillingDemoModelVersion) + if status != readBillingDemoStatusSuccess { + continue + } + for _, unit := range op.units { + metrics.AddReadBillingDemoBaseUnits(op.site, op.opClass, op.operatorKind, unit.unit, unit.source, unit.side, readBillingDemoModelVersion, unit.value) + metrics.ObserveReadBillingDemoRowWidth(op.site, op.opClass, op.operatorKind, unit.widthSource, readBillingDemoModelVersion, unit.rowWidth) + } + } +} + +func explainRUError(status explainRUStatus) error { + return errors.NewNoStackErrorf("EXPLAIN ANALYZE FORMAT='RU' is not supported for this target: %s", status) +} + +func recordExplainRUStatus(status explainRUStatus) { + metrics.RecordExplainRUStatus(string(status)) +} + +func (e *Explain) recordExplainRUStatus(status explainRUStatus) { + if e == nil || e.ruStatusRecorded { + return + } + e.ruStatusRecorded = true + recordExplainRUStatus(status) +} + +// explainRUSelectGateStatus is the first-demo pre-execution safety gate. It +// accepts only SELECT keyword surfaces and set operations whose leaves can be +// checked before EXPLAIN ANALYZE can run the target statement. +func explainRUSelectGateStatus(stmt ast.StmtNode) explainRUStatus { + switch x := stmt.(type) { + case *ast.SelectStmt: + return explainRUValidateSelectNode(x) + case *ast.SetOprStmt: + if x == nil || x.SelectList == nil { + return explainRUStatusUnsupportedNonSelect + } + return explainRUValidateSetOprSelectList(x.SelectList) + default: + return explainRUStatusUnsupportedNonSelect + } +} + +func explainRUValidateSetOprSelectList(list *ast.SetOprSelectList) explainRUStatus { + if list == nil || len(list.Selects) == 0 { + return explainRUStatusUnsupportedNonSelect + } + for _, sel := range list.Selects { + switch x := sel.(type) { + case *ast.SelectStmt: + if status := explainRUValidateSelectNode(x); status != explainRUStatusSuccess { + return status + } + case *ast.SetOprSelectList: + if status := explainRUValidateSetOprSelectList(x); status != explainRUStatusSuccess { + return status + } + default: + // SetOprSelectList is documented as SELECT/TABLE/VALUES capable. Fail + // closed until non-SELECT leaves have an explicit attribution design. + return explainRUStatusUnsupportedNonSelect + } + } + visitor := &explainRUSideEffectVisitor{status: explainRUStatusSuccess} + list.Accept(visitor) + return visitor.status +} + +func explainRUValidateSelectNode(sel *ast.SelectStmt) explainRUStatus { + if sel == nil || sel.Kind != ast.SelectStmtKindSelect { + return explainRUStatusUnsupportedNonSelect + } + if sel.SelectIntoOpt != nil { + return explainRUStatusUnsupportedSideEffecting + } + if sel.LockInfo != nil && sel.LockInfo.LockType != ast.SelectLockNone { + return explainRUStatusUnsupportedLockingSelect + } + visitor := &explainRUSideEffectVisitor{status: explainRUStatusSuccess} + sel.Accept(visitor) + return visitor.status +} + +type explainRUSideEffectVisitor struct { + status explainRUStatus +} + +func (v *explainRUSideEffectVisitor) Enter(n ast.Node) (ast.Node, bool) { + if v.status != explainRUStatusSuccess { + return n, true + } + switch x := n.(type) { + case *ast.SelectStmt: + if x.Kind != ast.SelectStmtKindSelect { + v.status = explainRUStatusUnsupportedNonSelect + return n, true + } + if x.SelectIntoOpt != nil { + v.status = explainRUStatusUnsupportedSideEffecting + return n, true + } + if x.LockInfo != nil && x.LockInfo.LockType != ast.SelectLockNone { + v.status = explainRUStatusUnsupportedLockingSelect + return n, true + } + case *ast.VariableExpr: + // User-variable assignment is syntactically a SELECT expression but + // mutates session state, so it is outside the side-effect-free demo scope. + if x.Value != nil { + v.status = explainRUStatusUnsupportedSideEffecting + return n, true + } + case *ast.FuncCallExpr: + if explainRUFuncCallHasSideEffect(x) { + v.status = explainRUStatusUnsupportedSideEffecting + return n, true + } + } + return n, false +} + +func (v *explainRUSideEffectVisitor) Leave(n ast.Node) (ast.Node, bool) { + return n, v.status == explainRUStatusSuccess +} + +func explainRUFuncCallHasSideEffect(fn *ast.FuncCallExpr) bool { + if fn == nil { + return false + } + switch strings.ToLower(fn.FnName.L) { + case ast.GetLock, ast.ReleaseLock, ast.ReleaseAllLocks, ast.NextVal, ast.SetVal, ast.Sleep: + return true + case ast.LastInsertId: + return len(fn.Args) > 0 + default: + return false + } +} + +func (e *Explain) renderRUExplain() (err error) { + start := time.Now() + status := explainRUStatusError + defer func() { + metrics.ObserveExplainRURenderDuration(string(status), time.Since(start).Seconds()) + e.recordExplainRUStatus(status) + }() + + if !e.Analyze { + status = explainRUStatusUnsupportedNonAnalyze + return explainRUError(explainRUStatusUnsupportedNonAnalyze) + } + if gateStatus := explainRUSelectGateStatus(e.ExecStmt); gateStatus != explainRUStatusSuccess { + status = gateStatus + return explainRUError(gateStatus) + } + flat := FlattenPhysicalPlan(e.TargetPlan, true) + if flat == nil || len(flat.Main) == 0 || flat.InExplain { + return errors.NewNoStackError("EXPLAIN ANALYZE FORMAT='RU' cannot render an empty target plan") + } + runtimeStats := e.RuntimeStatsColl + if runtimeStats == nil && e.SCtx() != nil && e.SCtx().GetSessionVars() != nil { + runtimeStats = e.SCtx().GetSessionVars().StmtCtx.RuntimeStatsColl + } + // The snapshot belongs to the target statement execution. Returning this + // EXPLAIN result can add more result-chunk counters later, so render output + // and Demo Metrics are derived from this frozen input and the generated rows. + _, snapshotStatus := explainRUExtractComponentSnapshot(runtimeStats, e.TargetPlan.ID()) + metrics.RecordExplainRUComponentSnapshot(string(snapshotStatus)) + result := buildReadBillingDemoResult(e.SCtx(), e.TargetPlan, e.ExecStmt, nil) + if result.status != readBillingDemoStatusSuccess { + operator := "" + if len(result.operators) > 0 { + op := result.operators[0] + operator = " operator=" + op.site + "/" + op.opClass + "/" + op.operatorKind + } + status = explainRUStatusError + return errors.NewNoStackErrorf( + "EXPLAIN ANALYZE FORMAT='RU' cannot render a complete read billing model result: status=%s reason=%s%s", + result.status, + result.reason, + operator, + ) + } + rows := explainRUBuildReadBillingRows(result, snapshotStatus) + + e.Rows = make([][]string, 0, len(rows)) + for _, row := range rows { + e.Rows = append(e.Rows, row.toStrings()) + explainRUObserveRow(row) + } + status = explainRUStatusSuccess + return nil +} + +func explainRUBuildReadBillingRows(result readBillingDemoResult, snapshotStatus explainRUComponentSnapshotStatus) []explainRURow { + rows := []explainRURow{{ + section: explainRUSectionSummary, + component: "total_preview_ru", + hasPreviewRU: true, + source: explainRUSourceSummaryTotal, + note: explainRUReadBillingSummaryNote(snapshotStatus), + }} + totalPreviewRU := 0.0 + for _, op := range result.operators { + if op.status != readBillingDemoStatusOperatorOK || !readBillingDemoOperatorBillable(op) { + continue + } + weights, hasWeights := readBillingDemoResolveWeights(op.site, op.opClass, readBillingDemoWeightVersion) + for _, unit := range op.units { + row := explainRUReadBillingUnitRow(op, unit) + if hasWeights { + if weight, previewRU, ok := readBillingDemoUnitPreviewRU(unit, weights); ok { + row.weight = weight + row.hasWeight = true + row.previewRU = previewRU + row.hasPreviewRU = true + totalPreviewRU += previewRU + } + } else { + row.note = appendExplainRUNote(row.note, "missing_weight") + } + rows = append(rows, row) + } + } + rows[0].previewRU = totalPreviewRU + return rows +} + +func explainRUReadBillingSummaryNote(snapshotStatus explainRUComponentSnapshotStatus) string { + note := "weight_version=" + readBillingDemoWeightVersion + if snapshotStatus != explainRUComponentSnapshotOK { + note = appendExplainRUNote(note, "component_snapshot_"+string(snapshotStatus)) + } + return note +} + +func explainRUReadBillingUnitRow(op readBillingDemoOperatorResult, unit readBillingDemoUnit) explainRURow { + row := explainRURow{ + section: explainRUSectionPlan, + id: op.id, + component: op.operatorKind, + operatorClass: op.site + "/" + op.opClass, + rowWidth: unit.rowWidth, + hasRowWidth: unit.rowWidth > 0, + rowWidthSource: unit.widthSource, + unit: unit.unit, + source: unit.source, + note: "input_side=" + unit.side + ",weight_version=" + readBillingDemoWeightVersion, + } + if op.hasActRows { + row.actRows = op.actRows + row.hasActRows = true + row.outputRows = op.actRows + row.hasOutputRows = true + } + switch unit.unit { + case readBillingDemoUnitFixedEvents: + row.count = int64(unit.value) + row.hasCount = true + case readBillingDemoUnitInputRows: + row.inputRows = int64(unit.value) + row.hasInputRows = true + row.workRows = int64(unit.value) + row.hasWorkRows = true + row.count = int64(unit.value) + row.hasCount = true + case readBillingDemoUnitInputBytes: + row.workBytes = unit.value + row.hasWorkBytes = true + } + return row +} + +func appendExplainRUNote(note, extra string) string { + if note == "" { + return extra + } + if extra == "" { + return note + } + return note + "," + extra +} + +func explainRUExtractComponentSnapshot(runtimeStats *execdetails.RuntimeStatsColl, targetPlanID int) (*execdetails.RURuntimeStats, explainRUComponentSnapshotStatus) { + // GetRootStats creates an empty entry for a missing plan ID; check + // ExistsRootStats first so "missing snapshot" stays observable. + if runtimeStats == nil || !runtimeStats.ExistsRootStats(targetPlanID) { + return nil, explainRUComponentSnapshotMissing + } + _, groups := runtimeStats.GetRootStats(targetPlanID).MergeStats() + for _, group := range groups { + ruStats, ok := group.(*execdetails.RURuntimeStats) + if !ok { + continue + } + if ruStats.RUVersion != rmclient.RUVersionV2 { + return ruStats, explainRUComponentSnapshotNonV2 + } + if ruStats.Metrics == nil { + return ruStats, explainRUComponentSnapshotNilMetrics + } + if ruStats.Metrics.Bypass() { + return ruStats, explainRUComponentSnapshotBypassed + } + return ruStats, explainRUComponentSnapshotOK + } + return nil, explainRUComponentSnapshotMissing +} + +func explainRURowWidth(sctx base.PlanContext, p base.Plan, includedRoot bool) (float64, string) { + if includedRoot { + // Root reader helpers describe SQL-visible row width for included TiDB + // work. Scan helpers below describe TiKV cop-side scan input width. + switch x := p.(type) { + case *physicalop.PhysicalTableReader: + if width := x.GetAvgRowSize(); width > 0 { + return width, explainRUWidthSourceOperatorHelper + } + case *physicalop.PhysicalIndexLookUpReader: + if width := x.GetAvgTableRowSize(); width > 0 { + return width, explainRUWidthSourceOperatorHelper + } + case *physicalop.PhysicalIndexMergeReader: + if width := x.GetAvgTableRowSize(); width > 0 { + return width, explainRUWidthSourceOperatorHelper + } + case *physicalop.PointGetPlan: + if width := x.GetAvgRowSize(); width > 0 { + return width, explainRUWidthSourceOperatorHelper + } + case *physicalop.BatchPointGetPlan: + if width := x.GetAvgRowSize(); width > 0 { + return width, explainRUWidthSourceOperatorHelper + } + } + } else { + switch x := p.(type) { + case *physicalop.PhysicalTableScan: + if width := x.GetScanRowSize(); width > 0 { + return width, explainRUWidthSourceOperatorHelper + } + case *physicalop.PhysicalIndexScan: + if width := x.GetScanRowSize(); width > 0 { + return width, explainRUWidthSourceOperatorHelper + } + } + } + if sctx == nil { + sctx = p.SCtx() + } + if stats := p.StatsInfo(); sctx != nil && stats != nil && stats.HistColl != nil && p.Schema() != nil { + if width := cardinality.GetAvgRowSize(sctx, stats.HistColl, p.Schema().Columns, false, false); width > 0 { + return width, explainRUWidthSourcePlanStats + } + } + if p.Schema() != nil { + width := 0 + for _, col := range p.Schema().Columns { + width += chunk.EstimateTypeWidth(col.GetStaticType()) + } + if width > 0 { + return float64(width), explainRUWidthSourceSchemaTypeWidth + } + if len(p.Schema().Columns) > 0 { + return float64(8 * len(p.Schema().Columns)), explainRUWidthSourceSchemaFallback + } + } + return 8, explainRUWidthSourceSchemaFallback +} + +func (row explainRURow) toStrings() []string { + return []string{ + row.section, + row.id, + row.component, + row.operatorClass, + formatOptionalInt(row.actRows, row.hasActRows), + formatOptionalInt(row.inputRows, row.hasInputRows), + formatOptionalInt(row.outputRows, row.hasOutputRows), + formatOptionalFloat(row.rowWidth, row.hasRowWidth), + row.rowWidthSource, + formatOptionalInt(row.workRows, row.hasWorkRows), + formatOptionalFloat(row.workBytes, row.hasWorkBytes), + row.unit, + formatOptionalInt(row.count, row.hasCount), + formatOptionalFloat(row.weight, row.hasWeight), + formatOptionalFloat(row.previewRU, row.hasPreviewRU), + row.source, + row.note, + } +} + +func formatOptionalInt(v int64, ok bool) string { + if !ok { + return "" + } + return strconv.FormatInt(v, 10) +} + +func formatOptionalFloat(v float64, ok bool) string { + if !ok { + return "" + } + return strconv.FormatFloat(v, 'f', 6, 64) +} + +func explainRUObserveRow(row explainRURow) { + // Metrics are emitted from rendered rows so the Prometheus view matches the + // SQL output and avoids reading live counters after render-side accounting. + previewRU := -1.0 + if row.hasPreviewRU { + previewRU = row.previewRU + } + workRows := -1.0 + if row.hasWorkRows { + workRows = float64(row.workRows) + } + workBytes := -1.0 + if row.hasWorkBytes { + workBytes = row.workBytes + } + rowWidth := -1.0 + if row.section == explainRUSectionPlan && row.hasRowWidth { + rowWidth = row.rowWidth + } + component, operator := explainRUMetricComponentOperator(row) + metrics.ObserveExplainRURow(row.section, component, operator, row.source, row.rowWidthSource, previewRU, workRows, workBytes, rowWidth) +} + +func explainRUMetricComponentOperator(row explainRURow) (component, operator string) { + switch row.section { + case explainRUSectionPlan: + return "", row.component + default: + return row.component, "" + } +} + +func explainRUUnsupportedFormatError(format string) error { + return errors.Errorf("'explain format=%v' cannot work without 'analyze', please use 'explain analyze format=%v'", format, format) +} + +func isExplainRUFormat(format string) bool { + return strings.ToLower(format) == types.ExplainFormatRU +} diff --git a/pkg/planner/core/logical_plan_builder.go b/pkg/planner/core/logical_plan_builder.go index d37b0d418f2ee..1c974681aabd3 100644 --- a/pkg/planner/core/logical_plan_builder.go +++ b/pkg/planner/core/logical_plan_builder.go @@ -5439,7 +5439,15 @@ func (b *PlanBuilder) buildMemTable(_ context.Context, dbName ast.CIStr, tableIn p.Extractor = &TableStorageStatsExtractor{} case infoschema.TableTiFlashTables, infoschema.TableTiFlashSegments, infoschema.TableTiFlashIndexes: p.Extractor = &TiFlashSystemTableExtractor{} - case infoschema.TableStatementsSummary, infoschema.TableStatementsSummaryHistory, infoschema.TableTiDBStatementsStats: + case infoschema.TableStatementsSummary, infoschema.TableStatementsSummaryHistory, infoschema.TableTiDBStatementsStats, + infoschema.TableStatementsSummaryReadBillingDemoBaseUnits, + infoschema.TableStatementsSummaryHistoryReadBillingDemoBaseUnits, + infoschema.TableStatementsSummaryReadBillingDemoStatus, + infoschema.TableStatementsSummaryHistoryReadBillingDemoStatus, + infoschema.ClusterTableStatementsSummaryReadBillingDemoBaseUnits, + infoschema.ClusterTableStatementsSummaryHistoryReadBillingDemoBaseUnits, + infoschema.ClusterTableStatementsSummaryReadBillingDemoStatus, + infoschema.ClusterTableStatementsSummaryHistoryReadBillingDemoStatus: p.Extractor = &StatementsSummaryExtractor{} case infoschema.TableTiKVRegionPeers: p.Extractor = &TikvRegionPeersExtractor{} diff --git a/pkg/planner/core/operator/logicalop/logical_mem_table.go b/pkg/planner/core/operator/logicalop/logical_mem_table.go index f9854098e301b..2aeb5260b0465 100644 --- a/pkg/planner/core/operator/logicalop/logical_mem_table.go +++ b/pkg/planner/core/operator/logicalop/logical_mem_table.go @@ -81,10 +81,18 @@ func (p *LogicalMemTable) PruneColumns(parentUsedCols []*expression.Column) (bas case infoschema.TableStatementsSummary, infoschema.TableStatementsSummaryHistory, infoschema.TableTiDBStatementsStats, + infoschema.TableStatementsSummaryReadBillingDemoBaseUnits, + infoschema.TableStatementsSummaryHistoryReadBillingDemoBaseUnits, + infoschema.TableStatementsSummaryReadBillingDemoStatus, + infoschema.TableStatementsSummaryHistoryReadBillingDemoStatus, infoschema.TableSlowQuery, infoschema.ClusterTableStatementsSummary, infoschema.ClusterTableStatementsSummaryHistory, infoschema.ClusterTableTiDBStatementsStats, + infoschema.ClusterTableStatementsSummaryReadBillingDemoBaseUnits, + infoschema.ClusterTableStatementsSummaryHistoryReadBillingDemoBaseUnits, + infoschema.ClusterTableStatementsSummaryReadBillingDemoStatus, + infoschema.ClusterTableStatementsSummaryHistoryReadBillingDemoStatus, infoschema.ClusterTableSlowLog, infoschema.TableTiDBTrx, infoschema.ClusterTableTiDBTrx, diff --git a/pkg/planner/core/pb_to_plan.go b/pkg/planner/core/pb_to_plan.go index e8588cc9fc047..659d5ac0d92f4 100644 --- a/pkg/planner/core/pb_to_plan.go +++ b/pkg/planner/core/pb_to_plan.go @@ -146,7 +146,11 @@ func (b *PBPlanBuilder) pbToTableScan(e *tipb.Executor) (base.PhysicalPlan, erro } } p.Extractor = extractor - case infoschema.ClusterTableStatementsSummary, infoschema.ClusterTableStatementsSummaryHistory: + case infoschema.ClusterTableStatementsSummary, infoschema.ClusterTableStatementsSummaryHistory, + infoschema.ClusterTableStatementsSummaryReadBillingDemoBaseUnits, + infoschema.ClusterTableStatementsSummaryHistoryReadBillingDemoBaseUnits, + infoschema.ClusterTableStatementsSummaryReadBillingDemoStatus, + infoschema.ClusterTableStatementsSummaryHistoryReadBillingDemoStatus: p.Extractor = &StatementsSummaryExtractor{} case infoschema.ClusterTableTiDBIndexUsage: p.Extractor = NewInfoSchemaTiDBIndexUsageExtractor() diff --git a/pkg/planner/core/planbuilder.go b/pkg/planner/core/planbuilder.go index efc9dbbe2498b..7bfa7bdb051b3 100644 --- a/pkg/planner/core/planbuilder.go +++ b/pkg/planner/core/planbuilder.go @@ -5712,6 +5712,18 @@ func (b *PlanBuilder) buildTrace(trace *ast.TraceStmt) (base.Plan, error) { func (b *PlanBuilder) buildExplainPlan(targetPlan base.Plan, format string, explainBriefBinary string, analyze, explore bool, execStmt ast.StmtNode, runtimeStats *execdetails.RuntimeStatsColl, sqlDigest, replayerFile string) (base.Plan, error) { format = strings.ToLower(format) + if format == types.ExplainFormatRU && !analyze { + recordExplainRUStatus(explainRUStatusUnsupportedNonAnalyze) + return nil, explainRUUnsupportedFormatError(format) + } + if format == types.ExplainFormatRU && analyze { + // Secondary guard for callers that already have a target plan. The main + // SQL path below rejects before optimization or no-delay execution. + if status := explainRUSelectGateStatus(execStmt); status != explainRUStatusSuccess { + recordExplainRUStatus(status) + return nil, explainRUError(status) + } + } if format == types.ExplainFormatTrueCardCost && !analyze { return nil, errors.Errorf("'explain format=%v' cannot work without 'analyze', please use 'explain analyze format=%v'", format, format) } @@ -5748,6 +5760,12 @@ func (b *PlanBuilder) buildExplainFor(explainFor *ast.ExplainForStmt) (base.Plan targetPlan, ok := processInfo.Plan.(base.Plan) explainForFormat := strings.ToLower(explainFor.Format) + // Check RU before the missing-plan branch so idle and running connections + // return the same unsupported status. + if explainForFormat == types.ExplainFormatRU { + recordExplainRUStatus(explainRUStatusUnsupportedForConnection) + return nil, errors.Errorf("explain format '%s' for connection is not supported now: %s", explainForFormat, explainRUStatusUnsupportedForConnection) + } if !ok || targetPlan == nil { return &Explain{Format: explainForFormat}, nil } @@ -5801,6 +5819,23 @@ func (b *PlanBuilder) buildExplain(ctx context.Context, explain *ast.ExplainStmt } explain.Stmt = hintedStmt } + if isExplainRUFormat(explain.Format) { + // This is the primary FORMAT='RU' safety boundary. It runs after plan + // digest expansion but before target optimization/execution, which keeps + // DML no-delay paths and side-effecting SELECT forms from running first. + if !explain.Analyze { + recordExplainRUStatus(explainRUStatusUnsupportedNonAnalyze) + return nil, explainRUUnsupportedFormatError(strings.ToLower(explain.Format)) + } + if explain.Explore { + recordExplainRUStatus(explainRUStatusUnsupportedNonSelect) + return nil, explainRUError(explainRUStatusUnsupportedNonSelect) + } + if status := explainRUSelectGateStatus(explain.Stmt); status != explainRUStatusSuccess { + recordExplainRUStatus(status) + return nil, explainRUError(status) + } + } sctx, err := AsSctx(b.ctx) if err != nil { diff --git a/pkg/session/BUILD.bazel b/pkg/session/BUILD.bazel index 3dbd454337571..f1013f3bff3bd 100644 --- a/pkg/session/BUILD.bazel +++ b/pkg/session/BUILD.bazel @@ -126,6 +126,8 @@ go_library( "//pkg/util/sli", "//pkg/util/sqlescape", "//pkg/util/sqlexec", + "//pkg/util/stmtsummary", + "//pkg/util/stmtsummary/v2:stmtsummary", "//pkg/util/syncutil", "//pkg/util/timeutil", "//pkg/util/topsql", diff --git a/pkg/session/session.go b/pkg/session/session.go index a414b90cd3941..9b32c7da5f6dc 100644 --- a/pkg/session/session.go +++ b/pkg/session/session.go @@ -136,6 +136,8 @@ import ( "github.com/pingcap/tidb/pkg/util/sli" "github.com/pingcap/tidb/pkg/util/sqlescape" "github.com/pingcap/tidb/pkg/util/sqlexec" + "github.com/pingcap/tidb/pkg/util/stmtsummary" + stmtsummaryv2 "github.com/pingcap/tidb/pkg/util/stmtsummary/v2" "github.com/pingcap/tidb/pkg/util/syncutil" "github.com/pingcap/tidb/pkg/util/timeutil" "github.com/pingcap/tidb/pkg/util/topsql" @@ -2424,10 +2426,16 @@ func (s *session) ExecuteStmt(ctx context.Context, stmtNode ast.StmtNode) (sqlex return rs, err } -func (s *session) executeStmtImpl(ctx context.Context, stmtNode ast.StmtNode) (sqlexec.RecordSet, error) { +func (s *session) executeStmtImpl(ctx context.Context, stmtNode ast.StmtNode) (recordSet sqlexec.RecordSet, err error) { r, ctx := tracing.StartRegionEx(ctx, "session.ExecuteStmt") defer r.End() ctx = execdetails.ContextWithMissingExecDetailsInitialized(ctx) + readBillingDemoHandledByStmt := false + defer func() { + if err != nil && !readBillingDemoHandledByStmt { + s.recordReadBillingDemoEarlyError(stmtNode, err) + } + }() if err := s.PrepareTxnCtx(ctx, stmtNode); err != nil { return nil, err @@ -2572,7 +2580,6 @@ func (s *session) executeStmtImpl(ctx context.Context, stmtNode ast.StmtNode) (s } var stmt *executor.ExecStmt - var err error { // Transform abstract syntax tree to a physical plan(stored in executor.ExecStmt). @@ -2678,7 +2685,6 @@ func (s *session) executeStmtImpl(ctx context.Context, stmtNode ast.StmtNode) (s }() } - var recordSet sqlexec.RecordSet if stmt.PsStmt != nil { // point plan short path ctx, prevTraceID := resetStmtTraceID(ctx, s) @@ -2711,6 +2717,7 @@ func (s *session) executeStmtImpl(ctx context.Context, stmtNode ast.StmtNode) (s s.setLastTxnInfoBeforeTxnEnd() s.txn.changeToInvalid() } else { + readBillingDemoHandledByStmt = true recordSet, err = runStmt(ctx, s, stmt) } @@ -2766,6 +2773,46 @@ func resolvePreparedStmt(stmt ast.StmtNode, vars *variable.SessionVars) (ast.Stm return prepareStmt.PreparedAst.Stmt, nil } +func (s *session) recordReadBillingDemoEarlyError(stmtNode ast.StmtNode, err error) { + if err == nil { + return + } + metricsStmt := stmtNode + if resolvedStmt, resolveErr := resolvePreparedStmt(stmtNode, s.sessionVars); resolveErr == nil && resolvedStmt != nil { + metricsStmt = resolvedStmt + } + readBillingDemoStats := plannercore.RecordReadBillingDemoForStatement(s, nil, metricsStmt, err) + if readBillingDemoStats.IsEmpty() || s.sessionVars == nil || s.sessionVars.StmtCtx == nil { + return + } + userString := "" + if s.sessionVars.User != nil { + userString = s.sessionVars.User.Username + } + stmtCtx := s.sessionVars.StmtCtx + if stmtCtx.StmtType == "" { + stmtCtx.StmtType = stmtctx.GetStmtLabel(context.Background(), metricsStmt) + } + normalizedSQL, digest := stmtCtx.SQLDigest() + digestString := "" + if digest != nil { + digestString = digest.String() + } + isInternalSQL := (s.sessionVars.InRestrictedSQL || len(userString) == 0) && !s.sessionVars.InExplainExplore + stmtsummaryv2.AddReadBillingDemoStatusOnly(&stmtsummary.StmtExecInfo{ + SchemaName: strings.ToLower(s.sessionVars.CurrentDB), + NormalizedSQL: normalizedSQL, + Digest: digestString, + User: userString, + StmtCtx: stmtCtx, + StartTime: s.sessionVars.StartTime, + IsInternal: isInternalSQL, + Succeed: false, + ResourceGroupName: stmtCtx.ResourceGroupName, + ReadBillingDemoStats: readBillingDemoStats, + }) +} + func shouldBypass(ctx context.Context, stmtNode ast.StmtNode, sessVars *variable.SessionVars) bool { switch kv.GetInternalSourceType(ctx) { case kv.InternalTxnOthers: @@ -2992,6 +3039,7 @@ func runStmt(ctx context.Context, se *session, s sqlexec.Statement) (rs sqlexec. origTxnCtx := sessVars.TxnCtx err = se.checkTxnAborted(s) if err != nil { + se.recordReadBillingDemoEarlyError(s.GetStmtNode(), err) return nil, err } if sessVars.TxnCtx.CouldRetry && !s.IsReadOnly(sessVars) { @@ -2999,6 +3047,7 @@ func runStmt(ctx context.Context, se *session, s sqlexec.Statement) (rs sqlexec. // otherwise, the stmt won't be add into stmt history, and also don't need check. // About `stmt-count-limit`, see more in https://docs.pingcap.com/tidb/stable/tidb-configuration-file#stmt-count-limit if err := checkStmtLimit(ctx, se, false); err != nil { + se.recordReadBillingDemoEarlyError(s.GetStmtNode(), err) return nil, err } } diff --git a/pkg/sessionctx/vardef/tidb_vars.go b/pkg/sessionctx/vardef/tidb_vars.go index cbe8896a9c9cd..fda9d39bf8d60 100644 --- a/pkg/sessionctx/vardef/tidb_vars.go +++ b/pkg/sessionctx/vardef/tidb_vars.go @@ -768,6 +768,9 @@ const ( // TiDBEnableCollectExecutionInfo indicates that whether execution info is collected. TiDBEnableCollectExecutionInfo = "tidb_enable_collect_execution_info" + // TiDBEnableReadBillingDemo enables the first-version read billing demo base-unit metrics. + TiDBEnableReadBillingDemo = "tidb_enable_read_billing_demo" + // TiDBExecutorConcurrency is used for controlling the concurrency of all types of executors. TiDBExecutorConcurrency = "tidb_executor_concurrency" @@ -1594,6 +1597,7 @@ const ( DefTiDBFoundInPlanCache = false DefTiDBFoundInBinding = false DefTiDBEnableCollectExecutionInfo = true + DefTiDBEnableReadBillingDemo = false DefTiDBAllowAutoRandExplicitInsert = false DefTiDBEnableClusteredIndex = ClusteredIndexDefModeOn DefTiDBRedactLog = Off diff --git a/pkg/sessionctx/variable/session.go b/pkg/sessionctx/variable/session.go index 718703e440cfd..a0816309e0619 100644 --- a/pkg/sessionctx/variable/session.go +++ b/pkg/sessionctx/variable/session.go @@ -1231,6 +1231,9 @@ type SessionVars struct { // AllowProjectionPushDown enables pushdown projection on TiKV. AllowProjectionPushDown bool + // EnableReadBillingDemo indicates whether read billing demo metrics are emitted for SELECT reads. + EnableReadBillingDemo bool + // EnableStrictNotNullCheck enables strict not-null check for single-row insert in non-strict mode. EnableStrictNotNullCheck bool @@ -2491,6 +2494,7 @@ func NewSessionVars(hctx HookContext) *SessionVars { OptOrderingIdxSelRatio: vardef.DefTiDBOptOrderingIdxSelRatio, RegardNULLAsPoint: vardef.DefTiDBRegardNULLAsPoint, AllowProjectionPushDown: vardef.DefOptEnableProjectionPushDown, + EnableReadBillingDemo: vardef.DefTiDBEnableReadBillingDemo, SkipMissingPartitionStats: vardef.DefTiDBSkipMissingPartitionStats, IndexLookUpPushDownPolicy: vardef.DefTiDBIndexLookUpPushDownPolicy, OptPartialOrderedIndexForTopN: vardef.DefTiDBOptPartialOrderedIndexForTopN, diff --git a/pkg/sessionctx/variable/sysvar.go b/pkg/sessionctx/variable/sysvar.go index cc4c97f600e67..843f91caeb6b1 100644 --- a/pkg/sessionctx/variable/sysvar.go +++ b/pkg/sessionctx/variable/sysvar.go @@ -609,6 +609,10 @@ var defaultSysVars = []*SysVar{ }, GetGlobal: func(_ context.Context, s *SessionVars) (string, error) { return BoolToOnOff(config.GetGlobalConfig().Instance.EnableCollectExecutionInfo.Load()), nil }}, + {Scope: vardef.ScopeGlobal | vardef.ScopeSession, Name: vardef.TiDBEnableReadBillingDemo, Value: BoolToOnOff(vardef.DefTiDBEnableReadBillingDemo), Type: vardef.TypeBool, SetSession: func(s *SessionVars, val string) error { + s.EnableReadBillingDemo = TiDBOptOn(val) + return nil + }}, {Scope: vardef.ScopeInstance, Name: vardef.PluginLoad, Value: "", ReadOnly: true, GetGlobal: func(_ context.Context, s *SessionVars) (string, error) { return config.GetGlobalConfig().Instance.PluginLoad, nil }}, diff --git a/pkg/types/explain_format.go b/pkg/types/explain_format.go index 67aabe4b27287..95a8dd07385c5 100644 --- a/pkg/types/explain_format.go +++ b/pkg/types/explain_format.go @@ -41,6 +41,8 @@ var ( ExplainFormatPlanCache = "plan_cache" // ExplainFormatPlanTree displays the plan in a tree structure format ExplainFormatPlanTree = "plan_tree" + // ExplainFormatRU displays read billing model attribution for EXPLAIN ANALYZE. + ExplainFormatRU = "ru" // ExplainFormats stores the valid formats for explain statement, used by validator. ExplainFormats = []string{ @@ -57,5 +59,6 @@ var ( ExplainFormatCostTrace, ExplainFormatPlanCache, ExplainFormatPlanTree, + ExplainFormatRU, } ) diff --git a/pkg/util/execdetails/runtime_stats.go b/pkg/util/execdetails/runtime_stats.go index 7dbf875c4faab..7ff46a5401b2f 100644 --- a/pkg/util/execdetails/runtime_stats.go +++ b/pkg/util/execdetails/runtime_stats.go @@ -231,6 +231,14 @@ func (crs *CopRuntimeStats) GetTasks() int32 { return int32(crs.stats.procTimes.size) } +// GetScanDetail returns a copy of the coprocessor scan detail collected for this plan. +func (crs *CopRuntimeStats) GetScanDetail() util.ScanDetail { + if crs == nil { + return util.ScanDetail{} + } + return crs.scanDetail +} + var zeroTimeDetail = util.TimeDetail{} func (crs *CopRuntimeStats) String() string { diff --git a/pkg/util/stmtsummary/BUILD.bazel b/pkg/util/stmtsummary/BUILD.bazel index f98a374797061..dc6bf9ad49e60 100644 --- a/pkg/util/stmtsummary/BUILD.bazel +++ b/pkg/util/stmtsummary/BUILD.bazel @@ -4,6 +4,7 @@ go_library( name = "stmtsummary", srcs = [ "evicted.go", + "read_billing.go", "reader.go", "statement_summary.go", ], @@ -36,11 +37,12 @@ go_test( srcs = [ "evicted_test.go", "main_test.go", + "read_billing_test.go", "statement_summary_test.go", ], embed = [":stmtsummary"], flaky = True, - shard_count = 28, + shard_count = 32, deps = [ "//pkg/meta/model", "//pkg/metrics", diff --git a/pkg/util/stmtsummary/evicted.go b/pkg/util/stmtsummary/evicted.go index fec9e5f9399a6..4c77021ac2529 100644 --- a/pkg/util/stmtsummary/evicted.go +++ b/pkg/util/stmtsummary/evicted.go @@ -390,6 +390,18 @@ func addInfo(addTo *stmtSummaryByDigestElement, addWith *stmtSummaryByDigestElem addTo.sumErrors += addWith.sumErrors addTo.StmtRUSummary.Merge(&addWith.StmtRUSummary) + var acceptedSummary ReadBillingDemoBaseUnitSummary + addTo.ReadBillingDemoBaseUnitAggs, addTo.ReadBillingDemoStatusAggs, acceptedSummary = MergeReadBillingDemoAggMaps( + addTo.ReadBillingDemoBaseUnitAggs, + addTo.ReadBillingDemoStatusAggs, + addWith.ReadBillingDemoBaseUnitAggs, + addWith.ReadBillingDemoStatusAggs, + ) + if len(addWith.ReadBillingDemoBaseUnitAggs) == 0 && len(addWith.ReadBillingDemoStatusAggs) == 0 { + addTo.ReadBillingDemoBaseUnitSummary.Merge(&addWith.ReadBillingDemoBaseUnitSummary) + } else { + addTo.ReadBillingDemoBaseUnitSummary.Add(&acceptedSummary) + } // resourceGroupName might not be inited because when it is a evicted item. addTo.resourceGroupName = addWith.resourceGroupName } diff --git a/pkg/util/stmtsummary/read_billing.go b/pkg/util/stmtsummary/read_billing.go new file mode 100644 index 0000000000000..c03c4022f6f62 --- /dev/null +++ b/pkg/util/stmtsummary/read_billing.go @@ -0,0 +1,696 @@ +// Copyright 2026 PingCAP, Inc. +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package stmtsummary + +import ( + "cmp" + "slices" +) + +const ( + // MaxReadBillingDemoBaseUnitKeysPerRecord bounds read billing dimensions per + // statement summary digest/window record. + MaxReadBillingDemoBaseUnitKeysPerRecord = 256 + // MaxReadBillingDemoStatusKeysPerRecord bounds non-reserved read billing + // status dimensions per statement summary digest/window record. + MaxReadBillingDemoStatusKeysPerRecord = 128 +) + +// Read billing demo column names used by INFORMATION_SCHEMA table definitions +// and statement-summary readers. +const ( + ReadBillingDemoModelVersionStr = "MODEL_VERSION" + ReadBillingDemoWeightVersionStr = "WEIGHT_VERSION" + ReadBillingDemoSiteStr = "SITE" + ReadBillingDemoOpClassStr = "OP_CLASS" + ReadBillingDemoOperatorKindStr = "OPERATOR_KIND" + ReadBillingDemoUnitStr = "UNIT" + ReadBillingDemoInputSourceStr = "INPUT_SOURCE" + ReadBillingDemoInputSideStr = "INPUT_SIDE" + ReadBillingDemoRowWidthSource = "ROW_WIDTH_SOURCE" + ReadBillingDemoValueStr = "VALUE" + ReadBillingDemoSampleCountStr = "SAMPLE_COUNT" + ReadBillingDemoRowWidthSumStr = "ROW_WIDTH_SUM" + ReadBillingDemoAvgRowWidthStr = "AVG_ROW_WIDTH" + ReadBillingDemoStatusStr = "STATUS" + ReadBillingDemoReasonStr = "REASON" + ReadBillingDemoCountStr = "COUNT" +) + +const ( + readBillingDemoSiteStatement = "statement" + readBillingDemoOpClassStatement = "statement" + readBillingDemoOperatorStatement = "statement" + readBillingDemoStatusUnknownInput = "unknown_input" + readBillingDemoReasonAggregation = "aggregation_overflow" + readBillingDemoReasonStatusAggregation = "status_aggregation_overflow" +) + +// ReadBillingDemoTableKind identifies which read billing demo table is being read. +type ReadBillingDemoTableKind int + +const ( + // ReadBillingDemoTableBaseUnits expands coefficient-free base-unit aggregates. + ReadBillingDemoTableBaseUnits ReadBillingDemoTableKind = iota + // ReadBillingDemoTableStatus expands statement/operator status aggregates. + ReadBillingDemoTableStatus +) + +// ReadBillingDemoStatementStats is the structured read billing demo snapshot +// attached to one statement execution before statement summary aggregation. +type ReadBillingDemoStatementStats struct { + ModelVersion string + WeightVersion string + Statuses []ReadBillingDemoStatusSample + BaseUnits []ReadBillingDemoBaseUnitSample + Totals ReadBillingDemoBaseUnitSummary +} + +// IsEmpty returns whether the snapshot contains no read billing data. +func (s *ReadBillingDemoStatementStats) IsEmpty() bool { + if s == nil { + return true + } + return len(s.Statuses) == 0 && len(s.BaseUnits) == 0 && + s.Totals.SumReadBillingDemoFixedEvents == 0 && + s.Totals.SumReadBillingDemoInputRows == 0 && + s.Totals.SumReadBillingDemoInputBytes == 0 +} + +// StatusOnly returns the status portion of a read billing snapshot. +func (s ReadBillingDemoStatementStats) StatusOnly() ReadBillingDemoStatementStats { + return ReadBillingDemoStatementStats{ + ModelVersion: s.ModelVersion, + WeightVersion: s.WeightVersion, + Statuses: append([]ReadBillingDemoStatusSample(nil), s.Statuses...), + } +} + +// ReadBillingDemoStatusOnlyExecInfo returns a shallow copy that is safe for the +// status-only statement summary path. +func ReadBillingDemoStatusOnlyExecInfo(sei *StmtExecInfo) (*StmtExecInfo, bool) { + if sei == nil { + return nil, false + } + statusOnly := sei.ReadBillingDemoStats.StatusOnly() + if statusOnly.IsEmpty() { + return nil, false + } + copied := *sei + copied.ReadBillingDemoStats = statusOnly + copied.ReadBillingDemoBaseUnits = ReadBillingDemoBaseUnitSummary{} + return &copied, true +} + +// ReadBillingDemoBaseUnitSample is a coefficient-free read billing unit sample. +type ReadBillingDemoBaseUnitSample struct { + ModelVersion string + WeightVersion string + Site string + OpClass string + OperatorKind string + Unit string + InputSource string + InputSide string + RowWidthSource string + Value float64 + RowWidth float64 +} + +// ReadBillingDemoStatusSample is a statement/operator read billing status sample. +type ReadBillingDemoStatusSample struct { + ModelVersion string + WeightVersion string + Site string + OpClass string + OperatorKind string + Status string + Reason string +} + +// ReadBillingDemoBaseUnitKey is the dimension key for statement-summary base-unit aggregation. +type ReadBillingDemoBaseUnitKey struct { + ModelVersion string + WeightVersion string + Site string + OpClass string + OperatorKind string + Unit string + InputSource string + InputSide string + RowWidthSource string +} + +// ReadBillingDemoBaseUnitAgg is the aggregate for one base-unit key. +type ReadBillingDemoBaseUnitAgg struct { + Value float64 + SampleCount uint64 + RowWidthSum float64 +} + +// ReadBillingDemoBaseUnitAggEntry is a JSON-stable base-unit aggregate entry. +type ReadBillingDemoBaseUnitAggEntry struct { + ModelVersion string `json:"model_version"` + WeightVersion string `json:"weight_version"` + Site string `json:"site"` + OpClass string `json:"op_class"` + OperatorKind string `json:"operator_kind"` + Unit string `json:"unit"` + InputSource string `json:"input_source"` + InputSide string `json:"input_side"` + RowWidthSource string `json:"row_width_source"` + Value float64 `json:"value"` + SampleCount uint64 `json:"sample_count"` + RowWidthSum float64 `json:"row_width_sum"` +} + +// ReadBillingDemoStatusKey is the dimension key for statement-summary status aggregation. +type ReadBillingDemoStatusKey struct { + ModelVersion string + WeightVersion string + Site string + OpClass string + OperatorKind string + Status string + Reason string +} + +// ReadBillingDemoStatusAgg is the aggregate for one status key. +type ReadBillingDemoStatusAgg struct { + Count uint64 +} + +// ReadBillingDemoStatusAggEntry is a JSON-stable status aggregate entry. +type ReadBillingDemoStatusAggEntry struct { + ModelVersion string `json:"model_version"` + WeightVersion string `json:"weight_version"` + Site string `json:"site"` + OpClass string `json:"op_class"` + OperatorKind string `json:"operator_kind"` + Status string `json:"status"` + Reason string `json:"reason"` + Count uint64 `json:"count"` +} + +func makeReadBillingDemoBaseUnitKey(sample ReadBillingDemoBaseUnitSample) ReadBillingDemoBaseUnitKey { + return ReadBillingDemoBaseUnitKey{ + ModelVersion: sample.ModelVersion, + WeightVersion: sample.WeightVersion, + Site: sample.Site, + OpClass: sample.OpClass, + OperatorKind: sample.OperatorKind, + Unit: sample.Unit, + InputSource: sample.InputSource, + InputSide: sample.InputSide, + RowWidthSource: sample.RowWidthSource, + } +} + +func makeReadBillingDemoStatusKey(sample ReadBillingDemoStatusSample) ReadBillingDemoStatusKey { + return ReadBillingDemoStatusKey{ + ModelVersion: sample.ModelVersion, + WeightVersion: sample.WeightVersion, + Site: sample.Site, + OpClass: sample.OpClass, + OperatorKind: sample.OperatorKind, + Status: sample.Status, + Reason: sample.Reason, + } +} + +func (e ReadBillingDemoBaseUnitAggEntry) key() ReadBillingDemoBaseUnitKey { + return ReadBillingDemoBaseUnitKey{ + ModelVersion: e.ModelVersion, + WeightVersion: e.WeightVersion, + Site: e.Site, + OpClass: e.OpClass, + OperatorKind: e.OperatorKind, + Unit: e.Unit, + InputSource: e.InputSource, + InputSide: e.InputSide, + RowWidthSource: e.RowWidthSource, + } +} + +func (e ReadBillingDemoStatusAggEntry) key() ReadBillingDemoStatusKey { + return ReadBillingDemoStatusKey{ + ModelVersion: e.ModelVersion, + WeightVersion: e.WeightVersion, + Site: e.Site, + OpClass: e.OpClass, + OperatorKind: e.OperatorKind, + Status: e.Status, + Reason: e.Reason, + } +} + +func (a *ReadBillingDemoBaseUnitAgg) addSample(sample ReadBillingDemoBaseUnitSample) { + a.Value += sample.Value + a.SampleCount++ + a.RowWidthSum += sample.RowWidth +} + +func (a *ReadBillingDemoBaseUnitAgg) addEntry(entry ReadBillingDemoBaseUnitAggEntry) { + a.Value += entry.Value + a.SampleCount += entry.SampleCount + a.RowWidthSum += entry.RowWidthSum +} + +func (a *ReadBillingDemoStatusAgg) addSample() { + a.Count++ +} + +func (a *ReadBillingDemoStatusAgg) addEntry(entry ReadBillingDemoStatusAggEntry) { + a.Count += entry.Count +} + +func (s *ReadBillingDemoBaseUnitSummary) addSample(sample ReadBillingDemoBaseUnitSample) { + switch sample.Unit { + case "fixed_events": + s.SumReadBillingDemoFixedEvents += sample.Value + case "input_rows": + s.SumReadBillingDemoInputRows += sample.Value + case "input_bytes": + s.SumReadBillingDemoInputBytes += sample.Value + } +} + +func (s *ReadBillingDemoBaseUnitSummary) addEntry(entry ReadBillingDemoBaseUnitAggEntry) { + switch entry.Unit { + case "fixed_events": + s.SumReadBillingDemoFixedEvents += entry.Value + case "input_rows": + s.SumReadBillingDemoInputRows += entry.Value + case "input_bytes": + s.SumReadBillingDemoInputBytes += entry.Value + } +} + +func readBillingDemoBaseUnitEntry(key ReadBillingDemoBaseUnitKey, agg ReadBillingDemoBaseUnitAgg) ReadBillingDemoBaseUnitAggEntry { + return ReadBillingDemoBaseUnitAggEntry{ + ModelVersion: key.ModelVersion, + WeightVersion: key.WeightVersion, + Site: key.Site, + OpClass: key.OpClass, + OperatorKind: key.OperatorKind, + Unit: key.Unit, + InputSource: key.InputSource, + InputSide: key.InputSide, + RowWidthSource: key.RowWidthSource, + Value: agg.Value, + SampleCount: agg.SampleCount, + RowWidthSum: agg.RowWidthSum, + } +} + +func readBillingDemoStatusEntry(key ReadBillingDemoStatusKey, agg ReadBillingDemoStatusAgg) ReadBillingDemoStatusAggEntry { + return ReadBillingDemoStatusAggEntry{ + ModelVersion: key.ModelVersion, + WeightVersion: key.WeightVersion, + Site: key.Site, + OpClass: key.OpClass, + OperatorKind: key.OperatorKind, + Status: key.Status, + Reason: key.Reason, + Count: agg.Count, + } +} + +// ReadBillingDemoBaseUnitEntriesFromMap converts map aggregates into deterministic entries. +func ReadBillingDemoBaseUnitEntriesFromMap(aggs map[ReadBillingDemoBaseUnitKey]ReadBillingDemoBaseUnitAgg) []ReadBillingDemoBaseUnitAggEntry { + entries := make([]ReadBillingDemoBaseUnitAggEntry, 0, len(aggs)) + for key, agg := range aggs { + entries = append(entries, readBillingDemoBaseUnitEntry(key, agg)) + } + sortReadBillingDemoBaseUnitEntries(entries) + return entries +} + +// ReadBillingDemoStatusEntriesFromMap converts map aggregates into deterministic entries. +func ReadBillingDemoStatusEntriesFromMap(aggs map[ReadBillingDemoStatusKey]ReadBillingDemoStatusAgg) []ReadBillingDemoStatusAggEntry { + entries := make([]ReadBillingDemoStatusAggEntry, 0, len(aggs)) + for key, agg := range aggs { + entries = append(entries, readBillingDemoStatusEntry(key, agg)) + } + sortReadBillingDemoStatusEntries(entries) + return entries +} + +// AddReadBillingDemoStatementStatsToMaps merges one statement snapshot into v1 maps. +func AddReadBillingDemoStatementStatsToMaps( + baseUnitAggs map[ReadBillingDemoBaseUnitKey]ReadBillingDemoBaseUnitAgg, + statusAggs map[ReadBillingDemoStatusKey]ReadBillingDemoStatusAgg, + stats *ReadBillingDemoStatementStats, +) (map[ReadBillingDemoBaseUnitKey]ReadBillingDemoBaseUnitAgg, map[ReadBillingDemoStatusKey]ReadBillingDemoStatusAgg, ReadBillingDemoBaseUnitSummary) { + var acceptedSummary ReadBillingDemoBaseUnitSummary + if stats == nil { + return baseUnitAggs, statusAggs, acceptedSummary + } + for _, status := range stats.Statuses { + statusAggs, _ = addReadBillingDemoStatusSampleToMap(statusAggs, status, false) + } + for _, sample := range stats.BaseUnits { + var overflow bool + baseUnitAggs, overflow = addReadBillingDemoBaseUnitSampleToMap(baseUnitAggs, sample) + if overflow { + statusAggs = addReadBillingDemoReservedStatusToMap(statusAggs, stats.ModelVersion, stats.WeightVersion, readBillingDemoReasonAggregation) + continue + } + acceptedSummary.addSample(sample) + } + return baseUnitAggs, statusAggs, acceptedSummary +} + +// MergeReadBillingDemoAggMaps merges source maps into destination maps with the same caps used on write. +func MergeReadBillingDemoAggMaps( + dstBase map[ReadBillingDemoBaseUnitKey]ReadBillingDemoBaseUnitAgg, + dstStatus map[ReadBillingDemoStatusKey]ReadBillingDemoStatusAgg, + srcBase map[ReadBillingDemoBaseUnitKey]ReadBillingDemoBaseUnitAgg, + srcStatus map[ReadBillingDemoStatusKey]ReadBillingDemoStatusAgg, +) (map[ReadBillingDemoBaseUnitKey]ReadBillingDemoBaseUnitAgg, map[ReadBillingDemoStatusKey]ReadBillingDemoStatusAgg, ReadBillingDemoBaseUnitSummary) { + var acceptedSummary ReadBillingDemoBaseUnitSummary + for key, agg := range srcStatus { + entry := readBillingDemoStatusEntry(key, agg) + dstStatus, _ = addReadBillingDemoStatusEntryToMap(dstStatus, entry, readBillingDemoStatusReasonReserved(entry.Reason)) + } + for key, agg := range srcBase { + var overflow bool + dstBase, overflow = addReadBillingDemoBaseUnitEntryToMap(dstBase, readBillingDemoBaseUnitEntry(key, agg)) + if overflow { + dstStatus = addReadBillingDemoReservedStatusToMap(dstStatus, key.ModelVersion, key.WeightVersion, readBillingDemoReasonAggregation) + continue + } + acceptedSummary.addEntry(readBillingDemoBaseUnitEntry(key, agg)) + } + return dstBase, dstStatus, acceptedSummary +} + +// AddReadBillingDemoStatementStatsToEntries merges one statement snapshot into v2 entries. +func AddReadBillingDemoStatementStatsToEntries( + baseUnitEntries []ReadBillingDemoBaseUnitAggEntry, + statusEntries []ReadBillingDemoStatusAggEntry, + stats *ReadBillingDemoStatementStats, +) ([]ReadBillingDemoBaseUnitAggEntry, []ReadBillingDemoStatusAggEntry, ReadBillingDemoBaseUnitSummary) { + var acceptedSummary ReadBillingDemoBaseUnitSummary + if stats == nil { + return baseUnitEntries, statusEntries, acceptedSummary + } + for _, status := range stats.Statuses { + statusEntries, _ = addReadBillingDemoStatusSampleToEntries(statusEntries, status, false) + } + for _, sample := range stats.BaseUnits { + var overflow bool + baseUnitEntries, overflow = addReadBillingDemoBaseUnitSampleToEntries(baseUnitEntries, sample) + if overflow { + statusEntries = addReadBillingDemoReservedStatusToEntries(statusEntries, stats.ModelVersion, stats.WeightVersion, readBillingDemoReasonAggregation) + continue + } + acceptedSummary.addSample(sample) + } + return baseUnitEntries, statusEntries, acceptedSummary +} + +// MergeReadBillingDemoEntrySlices merges source entries into destination entries with caps. +func MergeReadBillingDemoEntrySlices( + dstBase []ReadBillingDemoBaseUnitAggEntry, + dstStatus []ReadBillingDemoStatusAggEntry, + srcBase []ReadBillingDemoBaseUnitAggEntry, + srcStatus []ReadBillingDemoStatusAggEntry, +) ([]ReadBillingDemoBaseUnitAggEntry, []ReadBillingDemoStatusAggEntry, ReadBillingDemoBaseUnitSummary) { + var acceptedSummary ReadBillingDemoBaseUnitSummary + for _, entry := range srcStatus { + dstStatus, _ = addReadBillingDemoStatusEntryToEntries(dstStatus, entry, readBillingDemoStatusReasonReserved(entry.Reason)) + } + for _, entry := range srcBase { + var overflow bool + dstBase, overflow = addReadBillingDemoBaseUnitEntryToEntries(dstBase, entry) + if overflow { + dstStatus = addReadBillingDemoReservedStatusToEntries(dstStatus, entry.ModelVersion, entry.WeightVersion, readBillingDemoReasonAggregation) + continue + } + acceptedSummary.addEntry(entry) + } + return dstBase, dstStatus, acceptedSummary +} + +func addReadBillingDemoBaseUnitSampleToMap( + aggs map[ReadBillingDemoBaseUnitKey]ReadBillingDemoBaseUnitAgg, + sample ReadBillingDemoBaseUnitSample, +) (map[ReadBillingDemoBaseUnitKey]ReadBillingDemoBaseUnitAgg, bool) { + key := makeReadBillingDemoBaseUnitKey(sample) + if aggs == nil { + aggs = make(map[ReadBillingDemoBaseUnitKey]ReadBillingDemoBaseUnitAgg) + } + agg, exists := aggs[key] + if !exists && len(aggs) >= MaxReadBillingDemoBaseUnitKeysPerRecord { + return aggs, true + } + agg.addSample(sample) + aggs[key] = agg + return aggs, false +} + +func addReadBillingDemoBaseUnitEntryToMap( + aggs map[ReadBillingDemoBaseUnitKey]ReadBillingDemoBaseUnitAgg, + entry ReadBillingDemoBaseUnitAggEntry, +) (map[ReadBillingDemoBaseUnitKey]ReadBillingDemoBaseUnitAgg, bool) { + key := entry.key() + if aggs == nil { + aggs = make(map[ReadBillingDemoBaseUnitKey]ReadBillingDemoBaseUnitAgg) + } + agg, exists := aggs[key] + if !exists && len(aggs) >= MaxReadBillingDemoBaseUnitKeysPerRecord { + return aggs, true + } + agg.addEntry(entry) + aggs[key] = agg + return aggs, false +} + +func addReadBillingDemoStatusSampleToMap( + aggs map[ReadBillingDemoStatusKey]ReadBillingDemoStatusAgg, + sample ReadBillingDemoStatusSample, + reserved bool, +) (map[ReadBillingDemoStatusKey]ReadBillingDemoStatusAgg, bool) { + key := makeReadBillingDemoStatusKey(sample) + if aggs == nil { + aggs = make(map[ReadBillingDemoStatusKey]ReadBillingDemoStatusAgg) + } + agg, exists := aggs[key] + if !exists && !reserved && readBillingDemoNonReservedStatusKeyCount(aggs) >= MaxReadBillingDemoStatusKeysPerRecord { + aggs = addReadBillingDemoReservedStatusToMap(aggs, sample.ModelVersion, sample.WeightVersion, readBillingDemoReasonStatusAggregation) + return aggs, true + } + agg.addSample() + aggs[key] = agg + return aggs, false +} + +func addReadBillingDemoStatusEntryToMap( + aggs map[ReadBillingDemoStatusKey]ReadBillingDemoStatusAgg, + entry ReadBillingDemoStatusAggEntry, + reserved bool, +) (map[ReadBillingDemoStatusKey]ReadBillingDemoStatusAgg, bool) { + key := entry.key() + if aggs == nil { + aggs = make(map[ReadBillingDemoStatusKey]ReadBillingDemoStatusAgg) + } + agg, exists := aggs[key] + if !exists && !reserved && readBillingDemoNonReservedStatusKeyCount(aggs) >= MaxReadBillingDemoStatusKeysPerRecord { + aggs = addReadBillingDemoReservedStatusToMap(aggs, entry.ModelVersion, entry.WeightVersion, readBillingDemoReasonStatusAggregation) + return aggs, true + } + agg.addEntry(entry) + aggs[key] = agg + return aggs, false +} + +func addReadBillingDemoReservedStatusToMap( + aggs map[ReadBillingDemoStatusKey]ReadBillingDemoStatusAgg, + modelVersion string, + weightVersion string, + reason string, +) map[ReadBillingDemoStatusKey]ReadBillingDemoStatusAgg { + sample := readBillingDemoReservedStatusSample(modelVersion, weightVersion, reason) + aggs, _ = addReadBillingDemoStatusSampleToMap(aggs, sample, true) + return aggs +} + +func readBillingDemoNonReservedStatusKeyCount(aggs map[ReadBillingDemoStatusKey]ReadBillingDemoStatusAgg) int { + count := 0 + for key := range aggs { + if readBillingDemoStatusReasonReserved(key.Reason) { + continue + } + count++ + } + return count +} + +func addReadBillingDemoBaseUnitSampleToEntries( + entries []ReadBillingDemoBaseUnitAggEntry, + sample ReadBillingDemoBaseUnitSample, +) ([]ReadBillingDemoBaseUnitAggEntry, bool) { + key := makeReadBillingDemoBaseUnitKey(sample) + for i := range entries { + if entries[i].key() == key { + entries[i].Value += sample.Value + entries[i].SampleCount++ + entries[i].RowWidthSum += sample.RowWidth + return entries, false + } + } + if len(entries) >= MaxReadBillingDemoBaseUnitKeysPerRecord { + return entries, true + } + entries = append(entries, readBillingDemoBaseUnitEntry(key, ReadBillingDemoBaseUnitAgg{ + Value: sample.Value, + SampleCount: 1, + RowWidthSum: sample.RowWidth, + })) + sortReadBillingDemoBaseUnitEntries(entries) + return entries, false +} + +func addReadBillingDemoBaseUnitEntryToEntries( + entries []ReadBillingDemoBaseUnitAggEntry, + entry ReadBillingDemoBaseUnitAggEntry, +) ([]ReadBillingDemoBaseUnitAggEntry, bool) { + key := entry.key() + for i := range entries { + if entries[i].key() == key { + entries[i].Value += entry.Value + entries[i].SampleCount += entry.SampleCount + entries[i].RowWidthSum += entry.RowWidthSum + return entries, false + } + } + if len(entries) >= MaxReadBillingDemoBaseUnitKeysPerRecord { + return entries, true + } + entries = append(entries, entry) + sortReadBillingDemoBaseUnitEntries(entries) + return entries, false +} + +func addReadBillingDemoStatusSampleToEntries( + entries []ReadBillingDemoStatusAggEntry, + sample ReadBillingDemoStatusSample, + reserved bool, +) ([]ReadBillingDemoStatusAggEntry, bool) { + key := makeReadBillingDemoStatusKey(sample) + for i := range entries { + if entries[i].key() == key { + entries[i].Count++ + return entries, false + } + } + if !reserved && readBillingDemoNonReservedStatusEntryCount(entries) >= MaxReadBillingDemoStatusKeysPerRecord { + entries = addReadBillingDemoReservedStatusToEntries(entries, sample.ModelVersion, sample.WeightVersion, readBillingDemoReasonStatusAggregation) + return entries, true + } + entries = append(entries, readBillingDemoStatusEntry(key, ReadBillingDemoStatusAgg{Count: 1})) + sortReadBillingDemoStatusEntries(entries) + return entries, false +} + +func addReadBillingDemoStatusEntryToEntries( + entries []ReadBillingDemoStatusAggEntry, + entry ReadBillingDemoStatusAggEntry, + reserved bool, +) ([]ReadBillingDemoStatusAggEntry, bool) { + key := entry.key() + for i := range entries { + if entries[i].key() == key { + entries[i].Count += entry.Count + return entries, false + } + } + if !reserved && readBillingDemoNonReservedStatusEntryCount(entries) >= MaxReadBillingDemoStatusKeysPerRecord { + entries = addReadBillingDemoReservedStatusToEntries(entries, entry.ModelVersion, entry.WeightVersion, readBillingDemoReasonStatusAggregation) + return entries, true + } + entries = append(entries, entry) + sortReadBillingDemoStatusEntries(entries) + return entries, false +} + +func addReadBillingDemoReservedStatusToEntries( + entries []ReadBillingDemoStatusAggEntry, + modelVersion string, + weightVersion string, + reason string, +) []ReadBillingDemoStatusAggEntry { + sample := readBillingDemoReservedStatusSample(modelVersion, weightVersion, reason) + entries, _ = addReadBillingDemoStatusSampleToEntries(entries, sample, true) + return entries +} + +func readBillingDemoReservedStatusSample(modelVersion string, weightVersion string, reason string) ReadBillingDemoStatusSample { + return ReadBillingDemoStatusSample{ + ModelVersion: modelVersion, + WeightVersion: weightVersion, + Site: readBillingDemoSiteStatement, + OpClass: readBillingDemoOpClassStatement, + OperatorKind: readBillingDemoOperatorStatement, + Status: readBillingDemoStatusUnknownInput, + Reason: reason, + } +} + +func readBillingDemoNonReservedStatusEntryCount(entries []ReadBillingDemoStatusAggEntry) int { + count := 0 + for _, entry := range entries { + if readBillingDemoStatusReasonReserved(entry.Reason) { + continue + } + count++ + } + return count +} + +func readBillingDemoStatusReasonReserved(reason string) bool { + return reason == readBillingDemoReasonAggregation || reason == readBillingDemoReasonStatusAggregation +} + +func sortReadBillingDemoBaseUnitEntries(entries []ReadBillingDemoBaseUnitAggEntry) { + slices.SortFunc(entries, func(i, j ReadBillingDemoBaseUnitAggEntry) int { + return cmp.Or( + cmp.Compare(i.ModelVersion, j.ModelVersion), + cmp.Compare(i.WeightVersion, j.WeightVersion), + cmp.Compare(i.Site, j.Site), + cmp.Compare(i.OpClass, j.OpClass), + cmp.Compare(i.OperatorKind, j.OperatorKind), + cmp.Compare(i.Unit, j.Unit), + cmp.Compare(i.InputSource, j.InputSource), + cmp.Compare(i.InputSide, j.InputSide), + cmp.Compare(i.RowWidthSource, j.RowWidthSource), + ) + }) +} + +func sortReadBillingDemoStatusEntries(entries []ReadBillingDemoStatusAggEntry) { + slices.SortFunc(entries, func(i, j ReadBillingDemoStatusAggEntry) int { + return cmp.Or( + cmp.Compare(i.ModelVersion, j.ModelVersion), + cmp.Compare(i.WeightVersion, j.WeightVersion), + cmp.Compare(i.Site, j.Site), + cmp.Compare(i.OpClass, j.OpClass), + cmp.Compare(i.OperatorKind, j.OperatorKind), + cmp.Compare(i.Status, j.Status), + cmp.Compare(i.Reason, j.Reason), + ) + }) +} diff --git a/pkg/util/stmtsummary/read_billing_test.go b/pkg/util/stmtsummary/read_billing_test.go new file mode 100644 index 0000000000000..58804863f45d9 --- /dev/null +++ b/pkg/util/stmtsummary/read_billing_test.go @@ -0,0 +1,141 @@ +// Copyright 2026 PingCAP, Inc. +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package stmtsummary + +import ( + "fmt" + "testing" + + "github.com/stretchr/testify/require" +) + +func TestReadBillingDemoAggregationCaps(t *testing.T) { + stats := ReadBillingDemoStatementStats{ + ModelVersion: "v1", + WeightVersion: "v1", + } + for i := 0; i < MaxReadBillingDemoBaseUnitKeysPerRecord+2; i++ { + stats.BaseUnits = append(stats.BaseUnits, ReadBillingDemoBaseUnitSample{ + ModelVersion: "v1", + WeightVersion: "v1", + Site: "tidb", + OpClass: fmt.Sprintf("op_%03d", i), + OperatorKind: "projection", + Unit: "input_rows", + InputSource: "runtime_act_rows", + InputSide: "all", + RowWidthSource: "operator_helper", + Value: float64(i + 1), + RowWidth: 8, + }) + } + + baseAggs, statusAggs, acceptedSummary := AddReadBillingDemoStatementStatsToMaps(nil, nil, &stats) + require.Len(t, baseAggs, MaxReadBillingDemoBaseUnitKeysPerRecord) + require.Equal(t, uint64(2), requireReadBillingDemoStatusReason(t, ReadBillingDemoStatusEntriesFromMap(statusAggs), readBillingDemoReasonAggregation).Count) + require.Equal(t, float64(MaxReadBillingDemoBaseUnitKeysPerRecord*(MaxReadBillingDemoBaseUnitKeysPerRecord+1)/2), acceptedSummary.SumReadBillingDemoInputRows) + + baseEntries, statusEntries, acceptedSummary := AddReadBillingDemoStatementStatsToEntries(nil, nil, &stats) + require.Len(t, baseEntries, MaxReadBillingDemoBaseUnitKeysPerRecord) + require.Equal(t, uint64(2), requireReadBillingDemoStatusReason(t, statusEntries, readBillingDemoReasonAggregation).Count) + require.Equal(t, float64(MaxReadBillingDemoBaseUnitKeysPerRecord*(MaxReadBillingDemoBaseUnitKeysPerRecord+1)/2), acceptedSummary.SumReadBillingDemoInputRows) + + statusOnly := ReadBillingDemoStatementStats{ + ModelVersion: "v1", + WeightVersion: "v1", + } + for i := 0; i < MaxReadBillingDemoStatusKeysPerRecord+2; i++ { + statusOnly.Statuses = append(statusOnly.Statuses, ReadBillingDemoStatusSample{ + ModelVersion: "v1", + WeightVersion: "v1", + Site: "tidb", + OpClass: fmt.Sprintf("op_%03d", i), + OperatorKind: "projection", + Status: "unsupported", + Reason: "unsupported_operator", + }) + } + + _, statusAggs, acceptedSummary = AddReadBillingDemoStatementStatsToMaps(nil, nil, &statusOnly) + require.Equal(t, MaxReadBillingDemoStatusKeysPerRecord, readBillingDemoNonReservedStatusKeyCount(statusAggs)) + require.Equal(t, uint64(2), requireReadBillingDemoStatusReason(t, ReadBillingDemoStatusEntriesFromMap(statusAggs), readBillingDemoReasonStatusAggregation).Count) + require.Zero(t, acceptedSummary.SumReadBillingDemoFixedEvents) + require.Zero(t, acceptedSummary.SumReadBillingDemoInputRows) + require.Zero(t, acceptedSummary.SumReadBillingDemoInputBytes) + + _, statusEntries, acceptedSummary = AddReadBillingDemoStatementStatsToEntries(nil, nil, &statusOnly) + require.Equal(t, MaxReadBillingDemoStatusKeysPerRecord, readBillingDemoNonReservedStatusEntryCount(statusEntries)) + require.Equal(t, uint64(2), requireReadBillingDemoStatusReason(t, statusEntries, readBillingDemoReasonStatusAggregation).Count) + require.Zero(t, acceptedSummary.SumReadBillingDemoFixedEvents) + require.Zero(t, acceptedSummary.SumReadBillingDemoInputRows) + require.Zero(t, acceptedSummary.SumReadBillingDemoInputBytes) +} + +func TestReadBillingDemoReservedStatusMergeBypassesStatusCap(t *testing.T) { + fullStatusAggs := make(map[ReadBillingDemoStatusKey]ReadBillingDemoStatusAgg) + fullStatusEntries := make([]ReadBillingDemoStatusAggEntry, 0, MaxReadBillingDemoStatusKeysPerRecord) + for i := 0; i < MaxReadBillingDemoStatusKeysPerRecord; i++ { + key := ReadBillingDemoStatusKey{ + ModelVersion: "v1", + WeightVersion: "v1", + Site: "tidb", + OpClass: fmt.Sprintf("op_%03d", i), + OperatorKind: "projection", + Status: "unsupported", + Reason: "unsupported_operator", + } + fullStatusAggs[key] = ReadBillingDemoStatusAgg{Count: 1} + fullStatusEntries = append(fullStatusEntries, readBillingDemoStatusEntry(key, ReadBillingDemoStatusAgg{Count: 1})) + } + + srcStatusAggs := map[ReadBillingDemoStatusKey]ReadBillingDemoStatusAgg{ + makeReadBillingDemoStatusKey(readBillingDemoReservedStatusSample("v1", "v1", readBillingDemoReasonAggregation)): { + Count: 7, + }, + makeReadBillingDemoStatusKey(readBillingDemoReservedStatusSample("v1", "v1", readBillingDemoReasonStatusAggregation)): { + Count: 11, + }, + } + _, mergedStatusAggs, _ := MergeReadBillingDemoAggMaps(nil, fullStatusAggs, nil, srcStatusAggs) + require.Equal(t, MaxReadBillingDemoStatusKeysPerRecord, readBillingDemoNonReservedStatusKeyCount(mergedStatusAggs)) + require.Equal(t, uint64(7), requireReadBillingDemoStatusReason(t, ReadBillingDemoStatusEntriesFromMap(mergedStatusAggs), readBillingDemoReasonAggregation).Count) + require.Equal(t, uint64(11), requireReadBillingDemoStatusReason(t, ReadBillingDemoStatusEntriesFromMap(mergedStatusAggs), readBillingDemoReasonStatusAggregation).Count) + + srcStatusEntries := []ReadBillingDemoStatusAggEntry{ + readBillingDemoStatusEntry( + makeReadBillingDemoStatusKey(readBillingDemoReservedStatusSample("v1", "v1", readBillingDemoReasonAggregation)), + ReadBillingDemoStatusAgg{Count: 7}, + ), + readBillingDemoStatusEntry( + makeReadBillingDemoStatusKey(readBillingDemoReservedStatusSample("v1", "v1", readBillingDemoReasonStatusAggregation)), + ReadBillingDemoStatusAgg{Count: 11}, + ), + } + _, mergedStatusEntries, _ := MergeReadBillingDemoEntrySlices(nil, fullStatusEntries, nil, srcStatusEntries) + require.Equal(t, MaxReadBillingDemoStatusKeysPerRecord, readBillingDemoNonReservedStatusEntryCount(mergedStatusEntries)) + require.Equal(t, uint64(7), requireReadBillingDemoStatusReason(t, mergedStatusEntries, readBillingDemoReasonAggregation).Count) + require.Equal(t, uint64(11), requireReadBillingDemoStatusReason(t, mergedStatusEntries, readBillingDemoReasonStatusAggregation).Count) +} + +func requireReadBillingDemoStatusReason(t *testing.T, entries []ReadBillingDemoStatusAggEntry, reason string) ReadBillingDemoStatusAggEntry { + t.Helper() + for _, entry := range entries { + if entry.Reason == reason { + return entry + } + } + require.Failf(t, "missing read billing status reason", "reason=%s entries=%v", reason, entries) + return ReadBillingDemoStatusAggEntry{} +} diff --git a/pkg/util/stmtsummary/reader.go b/pkg/util/stmtsummary/reader.go index 90edc89cbbb33..c0f5ca66b5f8d 100644 --- a/pkg/util/stmtsummary/reader.go +++ b/pkg/util/stmtsummary/reader.go @@ -15,7 +15,6 @@ package stmtsummary import ( - "fmt" "strings" "time" @@ -56,10 +55,9 @@ func NewStmtSummaryReader(user *auth.UserIdentity, hasProcessPriv bool, cols []* reader.columnValueFactories = make([]columnValueFactory, len(reader.columns)) for i, col := range reader.columns { factory, ok := columnValueFactoryMap[col.Name.O] - if !ok { - panic(fmt.Sprintf("should never happen, should register new column %v into columnValueFactoryMap", col.Name.O)) + if ok { + reader.columnValueFactories[i] = factory } - reader.columnValueFactories[i] = factory } return reader } @@ -135,6 +133,236 @@ func (ssr *stmtSummaryReader) GetStmtSummaryHistoryRows() [][]types.Datum { return rows } +// GetReadBillingDemoCurrentRows gets current-window read billing demo rows. +func (ssr *stmtSummaryReader) GetReadBillingDemoCurrentRows(kind ReadBillingDemoTableKind) [][]types.Datum { + ssMap := ssr.ssMap + ssMap.Lock() + values := ssMap.summaryMap.Values() + beginTime := ssMap.beginTimeForCurInterval + other := ssMap.other + ssMap.Unlock() + + rows := make([][]types.Datum, 0, len(values)) + for _, value := range values { + ssbd := value.(*stmtSummaryByDigest) + if ssr.checker != nil && !ssr.checker.isDigestValid(ssbd.digest) { + continue + } + rows = append(rows, ssr.getReadBillingDemoCurrentRows(ssbd, beginTime, kind)...) + } + if ssr.checker == nil { + rows = append(rows, ssr.getReadBillingDemoEvictedOtherRows(other, kind)...) + } + return rows +} + +// GetReadBillingDemoHistoryRows gets history-window read billing demo rows. +func (ssr *stmtSummaryReader) GetReadBillingDemoHistoryRows(kind ReadBillingDemoTableKind) [][]types.Datum { + ssMap := ssr.ssMap + ssMap.Lock() + values := ssMap.summaryMap.Values() + other := ssMap.other + ssMap.Unlock() + + historySize := ssMap.historySize() + rows := make([][]types.Datum, 0, len(values)*historySize) + for _, value := range values { + ssbd := value.(*stmtSummaryByDigest) + ssElements := ssbd.collectHistorySummaries(ssr.checker, historySize) + for _, ssElement := range ssElements { + rows = append(rows, ssr.getReadBillingDemoElementRows(ssElement, ssbd, kind)...) + } + } + if ssr.checker == nil { + rows = append(rows, ssr.getReadBillingDemoEvictedOtherHistoryRows(other, historySize, kind)...) + } + return rows +} + +func (ssr *stmtSummaryReader) getReadBillingDemoCurrentRows(ssbd *stmtSummaryByDigest, beginTimeForCurInterval int64, kind ReadBillingDemoTableKind) [][]types.Datum { + var ssElement *stmtSummaryByDigestElement + + ssbd.Lock() + if ssbd.initialized && ssbd.history.Len() > 0 { + ssElement = ssbd.history.Back().Value.(*stmtSummaryByDigestElement) + } + ssbd.Unlock() + + if ssElement == nil || ssElement.beginTime < beginTimeForCurInterval { + return nil + } + return ssr.getReadBillingDemoElementRows(ssElement, ssbd, kind) +} + +func (ssr *stmtSummaryReader) getReadBillingDemoEvictedOtherRows(ssbde *stmtSummaryByDigestEvicted, kind ReadBillingDemoTableKind) [][]types.Datum { + var seElement *stmtSummaryByDigestEvictedElement + + ssbde.Lock() + if ssbde.history.Len() > 0 { + seElement = ssbde.history.Back().Value.(*stmtSummaryByDigestEvictedElement) + } + ssbde.Unlock() + + if seElement == nil { + return nil + } + return ssr.getReadBillingDemoElementRows(seElement.otherSummary, new(stmtSummaryByDigest), kind) +} + +func (ssr *stmtSummaryReader) getReadBillingDemoEvictedOtherHistoryRows(ssbde *stmtSummaryByDigestEvicted, historySize int, kind ReadBillingDemoTableKind) [][]types.Datum { + ssbde.Lock() + seElements := ssbde.collectHistorySummaries(historySize) + ssbde.Unlock() + rows := make([][]types.Datum, 0, len(seElements)) + + ssbd := new(stmtSummaryByDigest) + for _, seElement := range seElements { + rows = append(rows, ssr.getReadBillingDemoElementRows(seElement.otherSummary, ssbd, kind)...) + } + return rows +} + +func (ssr *stmtSummaryReader) getReadBillingDemoElementRows(ssElement *stmtSummaryByDigestElement, ssbd *stmtSummaryByDigest, kind ReadBillingDemoTableKind) [][]types.Datum { + ssElement.Lock() + defer ssElement.Unlock() + if !ssr.isAuthed(&ssElement.stmtSummaryStats) { + return nil + } + + switch kind { + case ReadBillingDemoTableBaseUnits: + entries := ReadBillingDemoBaseUnitEntriesFromMap(ssElement.ReadBillingDemoBaseUnitAggs) + rows := make([][]types.Datum, 0, len(entries)) + for _, entry := range entries { + rows = append(rows, ssr.readBillingDemoBaseUnitRow(ssElement, ssbd, entry)) + } + return rows + case ReadBillingDemoTableStatus: + entries := ReadBillingDemoStatusEntriesFromMap(ssElement.ReadBillingDemoStatusAggs) + rows := make([][]types.Datum, 0, len(entries)) + for _, entry := range entries { + rows = append(rows, ssr.readBillingDemoStatusRow(ssElement, ssbd, entry)) + } + return rows + default: + return nil + } +} + +func (ssr *stmtSummaryReader) readBillingDemoBaseUnitRow(ssElement *stmtSummaryByDigestElement, ssbd *stmtSummaryByDigest, entry ReadBillingDemoBaseUnitAggEntry) []types.Datum { + row := make([]types.Datum, len(ssr.columns)) + for i, col := range ssr.columns { + row[i] = types.NewDatum(ssr.readBillingDemoBaseUnitColumnValue(col.Name.O, ssElement, ssbd, entry)) + } + return row +} + +func (ssr *stmtSummaryReader) readBillingDemoStatusRow(ssElement *stmtSummaryByDigestElement, ssbd *stmtSummaryByDigest, entry ReadBillingDemoStatusAggEntry) []types.Datum { + row := make([]types.Datum, len(ssr.columns)) + for i, col := range ssr.columns { + row[i] = types.NewDatum(ssr.readBillingDemoStatusColumnValue(col.Name.O, ssElement, ssbd, entry)) + } + return row +} + +func (ssr *stmtSummaryReader) readBillingDemoCommonColumnValue(col string, ssElement *stmtSummaryByDigestElement, ssbd *stmtSummaryByDigest) (any, bool) { + switch col { + case ClusterTableInstanceColumnNameStr: + return ssr.instanceAddr, true + case SummaryBeginTimeStr: + beginTime := time.Unix(ssElement.beginTime, 0) + if beginTime.Location() != ssr.tz { + beginTime = beginTime.In(ssr.tz) + } + return types.NewTime(types.FromGoTime(beginTime), mysql.TypeTimestamp, 0), true + case SummaryEndTimeStr: + endTime := time.Unix(ssElement.endTime, 0) + if endTime.Location() != ssr.tz { + endTime = endTime.In(ssr.tz) + } + return types.NewTime(types.FromGoTime(endTime), mysql.TypeTimestamp, 0), true + case StmtTypeStr: + return ssbd.stmtType, true + case SchemaNameStr: + return convertEmptyToNil(ssbd.schemaName), true + case DigestStr: + return convertEmptyToNil(ssbd.digest), true + case DigestTextStr: + return ssbd.normalizedSQL, true + case PlanDigestStr: + return convertEmptyToNil(ssbd.planDigest), true + case ResourceGroupName: + return ssElement.resourceGroupName, true + default: + return nil, false + } +} + +func (ssr *stmtSummaryReader) readBillingDemoBaseUnitColumnValue(col string, ssElement *stmtSummaryByDigestElement, ssbd *stmtSummaryByDigest, entry ReadBillingDemoBaseUnitAggEntry) any { + if value, ok := ssr.readBillingDemoCommonColumnValue(col, ssElement, ssbd); ok { + return value + } + switch col { + case ReadBillingDemoModelVersionStr: + return entry.ModelVersion + case ReadBillingDemoWeightVersionStr: + return entry.WeightVersion + case ReadBillingDemoSiteStr: + return entry.Site + case ReadBillingDemoOpClassStr: + return entry.OpClass + case ReadBillingDemoOperatorKindStr: + return entry.OperatorKind + case ReadBillingDemoUnitStr: + return entry.Unit + case ReadBillingDemoInputSourceStr: + return entry.InputSource + case ReadBillingDemoInputSideStr: + return entry.InputSide + case ReadBillingDemoRowWidthSource: + return entry.RowWidthSource + case ReadBillingDemoValueStr: + return entry.Value + case ReadBillingDemoSampleCountStr: + return entry.SampleCount + case ReadBillingDemoRowWidthSumStr: + return entry.RowWidthSum + case ReadBillingDemoAvgRowWidthStr: + if entry.SampleCount == 0 { + return float64(0) + } + return entry.RowWidthSum / float64(entry.SampleCount) + default: + return nil + } +} + +func (ssr *stmtSummaryReader) readBillingDemoStatusColumnValue(col string, ssElement *stmtSummaryByDigestElement, ssbd *stmtSummaryByDigest, entry ReadBillingDemoStatusAggEntry) any { + if value, ok := ssr.readBillingDemoCommonColumnValue(col, ssElement, ssbd); ok { + return value + } + switch col { + case ReadBillingDemoModelVersionStr: + return entry.ModelVersion + case ReadBillingDemoWeightVersionStr: + return entry.WeightVersion + case ReadBillingDemoSiteStr: + return entry.Site + case ReadBillingDemoOpClassStr: + return entry.OpClass + case ReadBillingDemoOperatorKindStr: + return entry.OperatorKind + case ReadBillingDemoStatusStr: + return entry.Status + case ReadBillingDemoReasonStr: + return entry.Reason + case ReadBillingDemoCountStr: + return entry.Count + default: + return nil + } +} + func (ssr *stmtSummaryReader) SetChecker(checker *stmtSummaryChecker) { ssr.checker = checker } @@ -153,6 +381,9 @@ func (ssr *stmtSummaryReader) getStmtByDigestCumulativeRow(ssbd *stmtSummaryByDi if !ssr.isAuthed(&ssbd.cumulative) { return nil } + if ssbd.cumulative.isReadBillingDemoStatusOnly() { + return nil + } datums := make([]types.Datum, len(ssr.columnValueFactories)) for i, factory := range ssr.columnValueFactories { @@ -184,6 +415,9 @@ func (ssr *stmtSummaryReader) getStmtByDigestElementRow(ssElement *stmtSummaryBy if !ssr.isAuthed(&ssElement.stmtSummaryStats) { return nil } + if ssElement.stmtSummaryStats.isReadBillingDemoStatusOnly() { + return nil + } datums := make([]types.Datum, len(ssr.columnValueFactories)) for i, factory := range ssr.columnValueFactories { @@ -368,6 +602,9 @@ const ( MaxQueuedRcTimeStr = "MAX_QUEUED_RC_TIME" AvgRequestUnitV2Str = "AVG_REQUEST_UNIT_V2" MaxRequestUnitV2Str = "MAX_REQUEST_UNIT_V2" + SumReadBillingDemoFixedEventsStr = "SUM_READ_BILLING_DEMO_FIXED_EVENTS" + SumReadBillingDemoInputRowsStr = "SUM_READ_BILLING_DEMO_INPUT_ROWS" + SumReadBillingDemoInputBytesStr = "SUM_READ_BILLING_DEMO_INPUT_BYTES" ResourceGroupName = "RESOURCE_GROUP" SumUnpackedBytesSentTiKVTotalStr = "SUM_UNPACKED_BYTES_SENT_TIKV_TOTAL" SumUnpackedBytesReceivedTiKVTotalStr = "SUM_UNPACKED_BYTES_RECEIVED_TIKV_TOTAL" @@ -925,6 +1162,15 @@ var columnValueFactoryMap = map[string]columnValueFactory{ MaxRequestUnitV2Str: func(_ *stmtSummaryReader, _ *stmtSummaryByDigestElement, _ *stmtSummaryByDigest, ssStats *stmtSummaryStats) any { return ssStats.MaxRUV2 }, + SumReadBillingDemoFixedEventsStr: func(_ *stmtSummaryReader, _ *stmtSummaryByDigestElement, _ *stmtSummaryByDigest, ssStats *stmtSummaryStats) any { + return ssStats.SumReadBillingDemoFixedEvents + }, + SumReadBillingDemoInputRowsStr: func(_ *stmtSummaryReader, _ *stmtSummaryByDigestElement, _ *stmtSummaryByDigest, ssStats *stmtSummaryStats) any { + return ssStats.SumReadBillingDemoInputRows + }, + SumReadBillingDemoInputBytesStr: func(_ *stmtSummaryReader, _ *stmtSummaryByDigestElement, _ *stmtSummaryByDigest, ssStats *stmtSummaryStats) any { + return ssStats.SumReadBillingDemoInputBytes + }, ResourceGroupName: func(_ *stmtSummaryReader, _ *stmtSummaryByDigestElement, _ *stmtSummaryByDigest, ssStats *stmtSummaryStats) any { return ssStats.resourceGroupName }, diff --git a/pkg/util/stmtsummary/statement_summary.go b/pkg/util/stmtsummary/statement_summary.go index d81e1a72c2275..b74fc046b5184 100644 --- a/pkg/util/stmtsummary/statement_summary.go +++ b/pkg/util/stmtsummary/statement_summary.go @@ -254,6 +254,9 @@ type stmtSummaryStats struct { // request-units resourceGroupName string StmtRUSummary + ReadBillingDemoBaseUnitSummary + ReadBillingDemoBaseUnitAggs map[ReadBillingDemoBaseUnitKey]ReadBillingDemoBaseUnitAgg + ReadBillingDemoStatusAggs map[ReadBillingDemoStatusKey]ReadBillingDemoStatusAgg StmtNetworkTrafficSummary planCacheUnqualifiedCount int64 @@ -296,15 +299,17 @@ type StmtExecInfo struct { WriteSQLRespDuration time.Duration - ResultRows int64 - TiKVExecDetails *util.ExecDetails - Prepared bool - KeyspaceName string - KeyspaceID uint32 - ResourceGroupName string - RUDetail *util.RUDetails - TotalRUV2 float64 - CPUUsages ppcpuusage.CPUUsages + ResultRows int64 + TiKVExecDetails *util.ExecDetails + Prepared bool + KeyspaceName string + KeyspaceID uint32 + ResourceGroupName string + RUDetail *util.RUDetails + TotalRUV2 float64 + ReadBillingDemoBaseUnits ReadBillingDemoBaseUnitSummary + ReadBillingDemoStats ReadBillingDemoStatementStats + CPUUsages ppcpuusage.CPUUsages PlanCacheUnqualified string @@ -432,6 +437,75 @@ func (ssMap *stmtSummaryByDigestMap) AddStatement(sei *StmtExecInfo) { ssMap.updateMetricsLocked() } +// AddReadBillingDemoStatusOnly records read billing demo statuses that happen +// before the normal statement finish path can build a full StmtExecInfo. +func (ssMap *stmtSummaryByDigestMap) AddReadBillingDemoStatusOnly(sei *StmtExecInfo) { + var ok bool + sei, ok = ReadBillingDemoStatusOnlyExecInfo(sei) + if !ok { + return + } + now := time.Now().Unix() + failpoint.Inject("mockTimeForStatementsSummary", func(val failpoint.Value) { + if unixTimeStr, ok := val.(string); ok { + unixTime, err := strconv.ParseInt(unixTimeStr, 10, 64) + if err != nil { + panic(err.Error()) + } + now = unixTime + } + }) + + intervalSeconds := ssMap.refreshInterval() + historySize := 0 + if ssMap.historyEnabled() { + historySize = ssMap.historySize() + } + + key := StmtDigestKeyPool.Get().(*StmtDigestKey) + + ssMap.Lock() + defer ssMap.Unlock() + + userForKey := "" + if ssMap.optGroupByUser.Load() { + userForKey = sei.User + } + key.Init(sei.SchemaName, sei.Digest, sei.PrevSQLDigest, sei.PlanDigest, sei.ResourceGroupName, userForKey) + + if !ssMap.Enabled() { + StmtDigestKeyPool.Put(key) + return + } + if sei.IsInternal && !ssMap.EnabledInternal() { + StmtDigestKeyPool.Put(key) + return + } + + if ssMap.beginTimeForCurInterval+intervalSeconds <= now { + ssMap.beginTimeForCurInterval = now / intervalSeconds * intervalSeconds + ssMap.currentWindowEvictedCount = 0 + } + + beginTime := ssMap.beginTimeForCurInterval + value, exist := ssMap.summaryMap.Get(key) + var summary *stmtSummaryByDigest + if !exist { + summary = new(stmtSummaryByDigest) + ssMap.summaryMap.Put(key, summary) + } else { + summary = value.(*stmtSummaryByDigest) + } + summary.isInternal = summary.isInternal && sei.IsInternal + if summary != nil { + summary.addReadBillingDemoStatusOnly(sei, beginTime, intervalSeconds, historySize) + } + if exist { + StmtDigestKeyPool.Put(key) + } + ssMap.updateMetricsLocked() +} + // Clear removes all statement summaries. func (ssMap *stmtSummaryByDigestMap) Clear() { ssMap.Lock() @@ -692,6 +766,59 @@ func (ssbd *stmtSummaryByDigest) add(sei *StmtExecInfo, beginTime int64, interva } } +func (ssbd *stmtSummaryByDigest) initReadBillingDemoStatusOnly(sei *StmtExecInfo) { + ssbd.cumulative = *newReadBillingDemoStatusOnlyStats(sei) + ssbd.schemaName = sei.SchemaName + ssbd.digest = sei.Digest + ssbd.planDigest = sei.PlanDigest + if sei.StmtCtx != nil { + ssbd.stmtType = sei.StmtCtx.StmtType + } + ssbd.normalizedSQL = formatSQL(sei.NormalizedSQL) + ssbd.history = list.New() + ssbd.initialized = true +} + +func (ssbd *stmtSummaryByDigest) addReadBillingDemoStatusOnly(sei *StmtExecInfo, beginTime int64, intervalSeconds int64, historySize int) { + ssElement, isElementNew := func() (*stmtSummaryByDigestElement, bool) { + ssbd.Lock() + defer ssbd.Unlock() + + initializedNow := false + if !ssbd.initialized { + ssbd.initReadBillingDemoStatusOnly(sei) + initializedNow = true + } + if !initializedNow { + ssbd.cumulative.addReadBillingDemoStatementStats(sei.User, &sei.ReadBillingDemoStats) + } + + var ssElement *stmtSummaryByDigestElement + isElementNew := true + if ssbd.history.Len() > 0 { + lastElement := ssbd.history.Back().Value.(*stmtSummaryByDigestElement) + if lastElement.beginTime >= beginTime { + ssElement = lastElement + isElementNew = false + } else { + lastElement.onExpire(intervalSeconds) + } + } + if isElementNew { + ssElement = newReadBillingDemoStatusOnlyElement(sei, beginTime) + ssbd.history.PushBack(ssElement) + } + for ssbd.history.Len() > historySize && ssbd.history.Len() > 1 { + ssbd.history.Remove(ssbd.history.Front()) + } + return ssElement, isElementNew + }() + + if !isElementNew { + ssElement.addReadBillingDemoStatusOnly(sei) + } +} + // collectHistorySummaries puts at most `historySize` summaries to an array. func (ssbd *stmtSummaryByDigest) collectHistorySummaries(checker *stmtSummaryChecker, historySize int) []*stmtSummaryByDigestElement { ssbd.Lock() @@ -754,6 +881,28 @@ func newStmtSummaryStats(sei *StmtExecInfo) *stmtSummaryStats { } } +func newReadBillingDemoStatusOnlyStats(sei *StmtExecInfo) *stmtSummaryStats { + sei, ok := ReadBillingDemoStatusOnlyExecInfo(sei) + if !ok { + return &stmtSummaryStats{} + } + startTime := sei.StartTime + if startTime.IsZero() { + startTime = time.Now() + } + stats := &stmtSummaryStats{ + backoffTypes: make(map[string]int), + authUsers: make(map[string]struct{}), + minLatency: time.Duration(math.MaxInt64), + minResultRows: math.MaxInt64, + firstSeen: startTime, + lastSeen: startTime, + resourceGroupName: sei.ResourceGroupName, + } + stats.addReadBillingDemoStatementStats(sei.User, &sei.ReadBillingDemoStats) + return stats +} + func newStmtSummaryByDigestElement(sei *StmtExecInfo, beginTime int64, intervalSeconds int64, warningCount int, affectedRows uint64) *stmtSummaryByDigestElement { ssElement := &stmtSummaryByDigestElement{ beginTime: beginTime, @@ -763,6 +912,13 @@ func newStmtSummaryByDigestElement(sei *StmtExecInfo, beginTime int64, intervalS return ssElement } +func newReadBillingDemoStatusOnlyElement(sei *StmtExecInfo, beginTime int64) *stmtSummaryByDigestElement { + return &stmtSummaryByDigestElement{ + beginTime: beginTime, + stmtSummaryStats: *newReadBillingDemoStatusOnlyStats(sei), + } +} + // onExpire is called when this element expires to history. func (ssElement *stmtSummaryByDigestElement) onExpire(intervalSeconds int64) { ssElement.Lock() @@ -781,6 +937,35 @@ func (ssElement *stmtSummaryByDigestElement) onExpire(intervalSeconds int64) { } } +func (ssStats *stmtSummaryStats) addReadBillingDemoStatementStats(user string, stats *ReadBillingDemoStatementStats) { + if stats == nil { + return + } + if len(user) > 0 { + if ssStats.authUsers == nil { + ssStats.authUsers = make(map[string]struct{}) + } + ssStats.authUsers[user] = struct{}{} + } + var acceptedSummary ReadBillingDemoBaseUnitSummary + ssStats.ReadBillingDemoBaseUnitAggs, ssStats.ReadBillingDemoStatusAggs, acceptedSummary = AddReadBillingDemoStatementStatsToMaps( + ssStats.ReadBillingDemoBaseUnitAggs, + ssStats.ReadBillingDemoStatusAggs, + stats, + ) + ssStats.ReadBillingDemoBaseUnitSummary.Add(&acceptedSummary) +} + +func (ssStats *stmtSummaryStats) isReadBillingDemoStatusOnly() bool { + if ssStats == nil || ssStats.execCount != 0 { + return false + } + return len(ssStats.ReadBillingDemoBaseUnitAggs) > 0 || len(ssStats.ReadBillingDemoStatusAggs) > 0 || + ssStats.SumReadBillingDemoFixedEvents != 0 || + ssStats.SumReadBillingDemoInputRows != 0 || + ssStats.SumReadBillingDemoInputBytes != 0 +} + func (ssStats *stmtSummaryStats) add(sei *StmtExecInfo, warningCount int, affectedRows uint64) { // add user to auth users set if len(sei.User) > 0 { @@ -997,6 +1182,17 @@ func (ssStats *stmtSummaryStats) add(sei *StmtExecInfo, warningCount int, affect // request-units ssStats.StmtRUSummary.Add(sei.RUDetail, sei.TotalRUV2) + if !sei.ReadBillingDemoStats.IsEmpty() { + var acceptedSummary ReadBillingDemoBaseUnitSummary + ssStats.ReadBillingDemoBaseUnitAggs, ssStats.ReadBillingDemoStatusAggs, acceptedSummary = AddReadBillingDemoStatementStatsToMaps( + ssStats.ReadBillingDemoBaseUnitAggs, + ssStats.ReadBillingDemoStatusAggs, + &sei.ReadBillingDemoStats, + ) + ssStats.ReadBillingDemoBaseUnitSummary.Add(&acceptedSummary) + } else { + ssStats.ReadBillingDemoBaseUnitSummary.Add(&sei.ReadBillingDemoBaseUnits) + } ssStats.storageKV = sei.StmtCtx.IsTiKV.Load() ssStats.storageMPP = sei.StmtCtx.IsTiFlash.Load() @@ -1011,6 +1207,13 @@ func (ssElement *stmtSummaryByDigestElement) add(sei *StmtExecInfo, intervalSeco ssElement.stmtSummaryStats.add(sei, warningCount, affectedRows) } +func (ssElement *stmtSummaryByDigestElement) addReadBillingDemoStatusOnly(sei *StmtExecInfo) { + ssElement.Lock() + defer ssElement.Unlock() + + ssElement.stmtSummaryStats.addReadBillingDemoStatementStats(sei.User, &sei.ReadBillingDemoStats) +} + // Truncate SQL to maxSQLLength. func formatSQL(sql string) string { maxSQLLength := StmtSummaryByDigestMap.maxSQLLength() @@ -1141,6 +1344,28 @@ func (s *StmtRUSummary) Merge(other *StmtRUSummary) { } } +// ReadBillingDemoBaseUnitSummary is the read billing demo base-unit summary for each type of statements. +type ReadBillingDemoBaseUnitSummary struct { + SumReadBillingDemoFixedEvents float64 `json:"sum_read_billing_demo_fixed_events"` + SumReadBillingDemoInputRows float64 `json:"sum_read_billing_demo_input_rows"` + SumReadBillingDemoInputBytes float64 `json:"sum_read_billing_demo_input_bytes"` +} + +// Add adds one read billing demo base-unit sample to the summary. +func (s *ReadBillingDemoBaseUnitSummary) Add(other *ReadBillingDemoBaseUnitSummary) { + if other == nil { + return + } + s.SumReadBillingDemoFixedEvents += other.SumReadBillingDemoFixedEvents + s.SumReadBillingDemoInputRows += other.SumReadBillingDemoInputRows + s.SumReadBillingDemoInputBytes += other.SumReadBillingDemoInputBytes +} + +// Merge merges another read billing demo base-unit summary. +func (s *ReadBillingDemoBaseUnitSummary) Merge(other *ReadBillingDemoBaseUnitSummary) { + s.Add(other) +} + // StmtNetworkTrafficSummary is the network traffic summary for each type of statements. type StmtNetworkTrafficSummary struct { UnpackedBytesSentTiKVTotal int64 `json:"unpacked_bytes_send_tikv_total"` diff --git a/pkg/util/stmtsummary/statement_summary_test.go b/pkg/util/stmtsummary/statement_summary_test.go index bdf4fac921c78..5b15bd02f1568 100644 --- a/pkg/util/stmtsummary/statement_summary_test.go +++ b/pkg/util/stmtsummary/statement_summary_test.go @@ -1053,6 +1053,252 @@ func TestToDatum(t *testing.T) { match(t, datums[1], expectedEvictedDatum...) } +func TestReadBillingDemoBaseUnitsToDatum(t *testing.T) { + ssMap := newStmtSummaryByDigestMap() + now := time.Now().Unix() + // to disable expiration + ssMap.beginTimeForCurInterval = now + 60 + + stmtExecInfo1 := generateAnyExecInfo() + stmtExecInfo1.ReadBillingDemoBaseUnits = ReadBillingDemoBaseUnitSummary{ + SumReadBillingDemoFixedEvents: 2, + SumReadBillingDemoInputRows: 100, + SumReadBillingDemoInputBytes: 2048, + } + ssMap.AddStatement(stmtExecInfo1) + + stmtExecInfo2 := generateAnyExecInfo() + stmtExecInfo2.ReadBillingDemoBaseUnits = ReadBillingDemoBaseUnitSummary{ + SumReadBillingDemoFixedEvents: 3, + SumReadBillingDemoInputRows: 200, + SumReadBillingDemoInputBytes: 4096, + } + ssMap.AddStatement(stmtExecInfo2) + + key := &StmtDigestKey{} + key.Init(stmtExecInfo1.SchemaName, stmtExecInfo1.Digest, "", stmtExecInfo1.PlanDigest, stmtExecInfo1.ResourceGroupName, "") + value, ok := ssMap.summaryMap.Get(key) + require.True(t, ok) + ssElement := value.(*stmtSummaryByDigest).history.Back().Value.(*stmtSummaryByDigestElement) + require.Equal(t, 5.0, ssElement.SumReadBillingDemoFixedEvents) + require.Equal(t, 300.0, ssElement.SumReadBillingDemoInputRows) + require.Equal(t, 6144.0, ssElement.SumReadBillingDemoInputBytes) + + cols := []*model.ColumnInfo{ + {Name: ast.NewCIStr(SumReadBillingDemoFixedEventsStr)}, + {Name: ast.NewCIStr(SumReadBillingDemoInputRowsStr)}, + {Name: ast.NewCIStr(SumReadBillingDemoInputBytesStr)}, + } + reader := NewStmtSummaryReader(nil, true, cols, "", time.UTC) + reader.ssMap = ssMap + datums := reader.GetStmtSummaryCurrentRows() + require.Len(t, datums, 1) + match(t, datums[0], 5.0, 300.0, 6144.0) +} + +func TestReadBillingDemoStructuredRowsToDatum(t *testing.T) { + ssMap := newStmtSummaryByDigestMap() + now := time.Now().Unix() + // to disable expiration + ssMap.beginTimeForCurInterval = now + 60 + + stmtExecInfo := generateAnyExecInfo() + stmtExecInfo.ReadBillingDemoStats = ReadBillingDemoStatementStats{ + ModelVersion: "v1", + WeightVersion: "v1", + Statuses: []ReadBillingDemoStatusSample{ + { + ModelVersion: "v1", + WeightVersion: "v1", + Site: "statement", + OpClass: "statement", + OperatorKind: "statement", + Status: "success", + Reason: "none", + }, + { + ModelVersion: "v1", + WeightVersion: "v1", + Site: "tidb", + OpClass: "projection_eval", + OperatorKind: "projection", + Status: "ok", + Reason: "none", + }, + }, + BaseUnits: []ReadBillingDemoBaseUnitSample{ + { + ModelVersion: "v1", + WeightVersion: "v1", + Site: "tidb", + OpClass: "projection_eval", + OperatorKind: "projection", + Unit: "fixed_events", + InputSource: "runtime_act_rows", + InputSide: "all", + RowWidthSource: "operator_helper", + Value: 2, + RowWidth: 16, + }, + { + ModelVersion: "v1", + WeightVersion: "v1", + Site: "tidb", + OpClass: "projection_eval", + OperatorKind: "projection", + Unit: "fixed_events", + InputSource: "runtime_act_rows", + InputSide: "all", + RowWidthSource: "operator_helper", + Value: 3, + RowWidth: 24, + }, + { + ModelVersion: "v1", + WeightVersion: "v1", + Site: "tidb", + OpClass: "join_hash", + OperatorKind: "hashjoin", + Unit: "input_rows", + InputSource: "runtime_act_rows", + InputSide: "build", + RowWidthSource: "schema_type_width", + Value: 100, + RowWidth: 32, + }, + }, + Totals: ReadBillingDemoBaseUnitSummary{ + SumReadBillingDemoFixedEvents: 5, + SumReadBillingDemoInputRows: 100, + }, + } + ssMap.AddStatement(stmtExecInfo) + + statusOnlyInfo := generateAnyExecInfo() + statusOnlyInfo.Digest = "status_digest" + statusOnlyInfo.NormalizedSQL = "status_only_sql" + statusOnlyInfo.ReadBillingDemoStats = ReadBillingDemoStatementStats{ + ModelVersion: "v1", + WeightVersion: "v1", + Statuses: []ReadBillingDemoStatusSample{{ + ModelVersion: "v1", + WeightVersion: "v1", + Site: "statement", + OpClass: "statement", + OperatorKind: "statement", + Status: "error", + Reason: "statement_error", + }}, + BaseUnits: []ReadBillingDemoBaseUnitSample{{ + ModelVersion: "v1", + WeightVersion: "v1", + Site: "tidb", + OpClass: "projection_eval", + OperatorKind: "projection", + Unit: "fixed_events", + InputSource: "runtime_act_rows", + InputSide: "all", + RowWidthSource: "operator_helper", + Value: 999, + RowWidth: 999, + }}, + Totals: ReadBillingDemoBaseUnitSummary{ + SumReadBillingDemoFixedEvents: 999, + }, + } + ssMap.AddReadBillingDemoStatusOnly(statusOnlyInfo) + + unknownInputInfo := generateAnyExecInfo() + unknownInputInfo.Digest = "unknown_input_digest" + unknownInputInfo.NormalizedSQL = "unknown_input_sql" + unknownInputInfo.ReadBillingDemoStats = ReadBillingDemoStatementStats{ + ModelVersion: "v1", + WeightVersion: "v1", + Statuses: []ReadBillingDemoStatusSample{{ + ModelVersion: "v1", + WeightVersion: "v1", + Site: "statement", + OpClass: "statement", + OperatorKind: "statement", + Status: "unknown_input", + Reason: "missing_runtime_stats", + }}, + } + ssMap.AddReadBillingDemoStatusOnly(unknownInputInfo) + + baseUnitCols := []*model.ColumnInfo{ + {Name: ast.NewCIStr(ReadBillingDemoSiteStr)}, + {Name: ast.NewCIStr(ReadBillingDemoOpClassStr)}, + {Name: ast.NewCIStr(ReadBillingDemoOperatorKindStr)}, + {Name: ast.NewCIStr(ReadBillingDemoUnitStr)}, + {Name: ast.NewCIStr(ReadBillingDemoInputSourceStr)}, + {Name: ast.NewCIStr(ReadBillingDemoInputSideStr)}, + {Name: ast.NewCIStr(ReadBillingDemoRowWidthSource)}, + {Name: ast.NewCIStr(ReadBillingDemoValueStr)}, + {Name: ast.NewCIStr(ReadBillingDemoSampleCountStr)}, + {Name: ast.NewCIStr(ReadBillingDemoRowWidthSumStr)}, + {Name: ast.NewCIStr(ReadBillingDemoAvgRowWidthStr)}, + } + reader := NewStmtSummaryReader(nil, true, baseUnitCols, "", time.UTC) + reader.ssMap = ssMap + baseRows := reader.GetReadBillingDemoCurrentRows(ReadBillingDemoTableBaseUnits) + require.Len(t, baseRows, 2) + expectedBaseRows := map[string]string{ + "tidb/join_hash/hashjoin/input_rows/runtime_act_rows/build/schema_type_width": "100 1 32 32", + "tidb/projection_eval/projection/fixed_events/runtime_act_rows/all/operator_helper": "5 2 40 20", + } + require.Equal(t, expectedBaseRows, readBillingDemoRowsByKey(baseRows, 7)) + require.Equal(t, expectedBaseRows, readBillingDemoRowsByKey(reader.GetReadBillingDemoHistoryRows(ReadBillingDemoTableBaseUnits), 7)) + + statusCols := []*model.ColumnInfo{ + {Name: ast.NewCIStr(DigestTextStr)}, + {Name: ast.NewCIStr(ReadBillingDemoSiteStr)}, + {Name: ast.NewCIStr(ReadBillingDemoOpClassStr)}, + {Name: ast.NewCIStr(ReadBillingDemoOperatorKindStr)}, + {Name: ast.NewCIStr(ReadBillingDemoStatusStr)}, + {Name: ast.NewCIStr(ReadBillingDemoReasonStr)}, + {Name: ast.NewCIStr(ReadBillingDemoCountStr)}, + } + statusReader := NewStmtSummaryReader(nil, true, statusCols, "", time.UTC) + statusReader.ssMap = ssMap + statusRows := statusReader.GetReadBillingDemoCurrentRows(ReadBillingDemoTableStatus) + require.Len(t, statusRows, 4) + expectedStatusRows := map[string]string{ + "normalized_sql/statement/statement/statement/success/none": "1", + "normalized_sql/tidb/projection_eval/projection/ok/none": "1", + "status_only_sql/statement/statement/statement/error/statement_error": "1", + "unknown_input_sql/statement/statement/statement/unknown_input/missing_runtime_stats": "1", + } + require.Equal(t, expectedStatusRows, readBillingDemoRowsByKey(statusRows, 6)) + require.Equal(t, expectedStatusRows, readBillingDemoRowsByKey(statusReader.GetReadBillingDemoHistoryRows(ReadBillingDemoTableStatus), 6)) + + normalCols := []*model.ColumnInfo{ + {Name: ast.NewCIStr(DigestStr)}, + {Name: ast.NewCIStr(ExecCountStr)}, + } + normalReader := NewStmtSummaryReader(nil, true, normalCols, "", time.UTC) + normalReader.ssMap = ssMap + normalRows := normalReader.GetStmtSummaryCurrentRows() + require.Len(t, normalRows, 1) + match(t, normalRows[0], "digest", 1) +} + +func readBillingDemoRowsByKey(rows [][]types.Datum, keyColumns int) map[string]string { + result := make(map[string]string, len(rows)) + for _, row := range rows { + keyParts := make([]string, 0, keyColumns) + for i := 0; i < keyColumns; i++ { + keyParts = append(keyParts, fmt.Sprintf("%v", row[i].GetValue())) + } + valueParts := make([]string, 0, len(row)-keyColumns) + for i := keyColumns; i < len(row); i++ { + valueParts = append(valueParts, fmt.Sprintf("%v", row[i].GetValue())) + } + result[strings.Join(keyParts, "/")] = strings.Join(valueParts, " ") + } + return result +} + // Test AddStatement and ToDatum parallel. func TestAddStatementParallel(t *testing.T) { ssMap := newStmtSummaryByDigestMap() diff --git a/pkg/util/stmtsummary/v2/BUILD.bazel b/pkg/util/stmtsummary/v2/BUILD.bazel index 2ad5781b7b0b9..a8220186c06e5 100644 --- a/pkg/util/stmtsummary/v2/BUILD.bazel +++ b/pkg/util/stmtsummary/v2/BUILD.bazel @@ -49,7 +49,7 @@ go_test( ], embed = [":stmtsummary"], flaky = True, - shard_count = 18, + shard_count = 23, deps = [ "//pkg/config", "//pkg/meta/model", diff --git a/pkg/util/stmtsummary/v2/column.go b/pkg/util/stmtsummary/v2/column.go index 68ec4e170a88e..2ada88544d364 100644 --- a/pkg/util/stmtsummary/v2/column.go +++ b/pkg/util/stmtsummary/v2/column.go @@ -144,6 +144,9 @@ const ( MaxQueuedRcTimeStr = "MAX_QUEUED_RC_TIME" AvgRequestUnitV2 = "AVG_REQUEST_UNIT_V2" MaxRequestUnitV2 = "MAX_REQUEST_UNIT_V2" + SumReadBillingDemoFixedEventsStr = "SUM_READ_BILLING_DEMO_FIXED_EVENTS" + SumReadBillingDemoInputRowsStr = "SUM_READ_BILLING_DEMO_INPUT_ROWS" + SumReadBillingDemoInputBytesStr = "SUM_READ_BILLING_DEMO_INPUT_BYTES" ResourceGroupName = "RESOURCE_GROUP" SumUnpackedBytesSentTiKVTotalStr = "SUM_UNPACKED_BYTES_SENT_TIKV_TOTAL" SumUnpackedBytesReceivedTiKVTotalStr = "SUM_UNPACKED_BYTES_RECEIVED_TIKV_TOTAL" @@ -523,6 +526,15 @@ var columnFactoryMap = map[string]columnFactory{ MaxRequestUnitV2: func(_ columnInfo, record *StmtRecord) any { return record.MaxRUV2 }, + SumReadBillingDemoFixedEventsStr: func(_ columnInfo, record *StmtRecord) any { + return record.SumReadBillingDemoFixedEvents + }, + SumReadBillingDemoInputRowsStr: func(_ columnInfo, record *StmtRecord) any { + return record.SumReadBillingDemoInputRows + }, + SumReadBillingDemoInputBytesStr: func(_ columnInfo, record *StmtRecord) any { + return record.SumReadBillingDemoInputBytes + }, ResourceGroupName: func(_ columnInfo, record *StmtRecord) any { return record.ResourceGroupName }, diff --git a/pkg/util/stmtsummary/v2/column_test.go b/pkg/util/stmtsummary/v2/column_test.go index dd07fd5e22077..6431beaabde3b 100644 --- a/pkg/util/stmtsummary/v2/column_test.go +++ b/pkg/util/stmtsummary/v2/column_test.go @@ -37,6 +37,9 @@ func TestColumn(t *testing.T) { {Name: ast.NewCIStr(ExecCountStr)}, {Name: ast.NewCIStr(SumLatencyStr)}, {Name: ast.NewCIStr(MaxLatencyStr)}, + {Name: ast.NewCIStr(SumReadBillingDemoFixedEventsStr)}, + {Name: ast.NewCIStr(SumReadBillingDemoInputRowsStr)}, + {Name: ast.NewCIStr(SumReadBillingDemoInputBytesStr)}, {Name: ast.NewCIStr(AvgTidbCPUTimeStr)}, {Name: ast.NewCIStr(AvgTikvCPUTimeStr)}, } @@ -69,6 +72,12 @@ func TestColumn(t *testing.T) { require.Equal(t, int64(record.SumLatency), column) case MaxLatencyStr: require.Equal(t, int64(record.MaxLatency), column) + case SumReadBillingDemoFixedEventsStr: + require.Equal(t, record.SumReadBillingDemoFixedEvents, column) + case SumReadBillingDemoInputRowsStr: + require.Equal(t, record.SumReadBillingDemoInputRows, column) + case SumReadBillingDemoInputBytesStr: + require.Equal(t, record.SumReadBillingDemoInputBytes, column) case AvgTidbCPUTimeStr: require.Equal(t, int64(record.SumTidbCPU), column) case AvgTikvCPUTimeStr: diff --git a/pkg/util/stmtsummary/v2/logger.go b/pkg/util/stmtsummary/v2/logger.go index 028b20ed11fc9..c37956f9bd5ab 100644 --- a/pkg/util/stmtsummary/v2/logger.go +++ b/pkg/util/stmtsummary/v2/logger.go @@ -61,7 +61,7 @@ func (s *stmtLogStorage) persist(w *stmtWindow, end time.Time) { r.Unlock() } w.evicted.Lock() - if w.evicted.otherForPersist.ExecCount > 0 { + if w.evicted.otherForPersist.ExecCount > 0 || w.evicted.otherForPersist.hasReadBillingDemoData() { w.evicted.otherForPersist.Begin = begin w.evicted.otherForPersist.End = end.Unix() s.log(w.evicted.otherForPersist) diff --git a/pkg/util/stmtsummary/v2/reader.go b/pkg/util/stmtsummary/v2/reader.go index 589fc4aa0a59f..44fa1205b6d38 100644 --- a/pkg/util/stmtsummary/v2/reader.go +++ b/pkg/util/stmtsummary/v2/reader.go @@ -31,10 +31,12 @@ import ( "github.com/pingcap/tidb/pkg/config" "github.com/pingcap/tidb/pkg/meta/model" "github.com/pingcap/tidb/pkg/parser/auth" + "github.com/pingcap/tidb/pkg/parser/mysql" "github.com/pingcap/tidb/pkg/types" "github.com/pingcap/tidb/pkg/util" "github.com/pingcap/tidb/pkg/util/logutil" "github.com/pingcap/tidb/pkg/util/set" + stmtsummarybase "github.com/pingcap/tidb/pkg/util/stmtsummary" "go.uber.org/zap" ) @@ -63,6 +65,17 @@ type MemReader struct { checker *stmtChecker } +// ReadBillingDemoMemReader expands read billing demo aggregates from the +// current in-memory statement summary window. +type ReadBillingDemoMemReader struct { + s *StmtSummary + columns []*model.ColumnInfo + instanceAddr string + timeLocation *time.Location + kind stmtsummarybase.ReadBillingDemoTableKind + checker *stmtChecker +} + // NewMemReader creates a MemReader from StmtSummary and other necessary parameters. func NewMemReader(s *StmtSummary, columns []*model.ColumnInfo, @@ -87,6 +100,31 @@ func NewMemReader(s *StmtSummary, } } +// NewReadBillingDemoMemReader creates a reader for current read billing demo rows. +func NewReadBillingDemoMemReader(s *StmtSummary, + columns []*model.ColumnInfo, + instanceAddr string, + timeLocation *time.Location, + user *auth.UserIdentity, + hasProcessPriv bool, + digests set.StringSet, + timeRanges []*StmtTimeRange, + kind stmtsummarybase.ReadBillingDemoTableKind) *ReadBillingDemoMemReader { + return &ReadBillingDemoMemReader{ + s: s, + columns: columns, + instanceAddr: instanceAddr, + timeLocation: timeLocation, + kind: kind, + checker: &stmtChecker{ + user: user, + hasProcessPriv: hasProcessPriv, + digests: digests, + timeRanges: timeRanges, + }, + } +} + // Rows returns rows converted from the current window's data maintained // in memory by StmtSummary. All evicted data will be aggregated into a // single row appended at the end. @@ -116,6 +154,9 @@ func (r *MemReader) Rows() [][]types.Datum { if !r.checker.hasPrivilege(record.AuthUsers) { return } + if record.StmtRecord.isReadBillingDemoStatusOnly() { + return + } record.Begin = w.begin.Unix() record.End = end row := make([]types.Datum, len(r.columnFactories)) @@ -147,6 +188,53 @@ func (r *MemReader) Rows() [][]types.Datum { return rows } +// Rows returns expanded read billing demo rows from the current window. +func (r *ReadBillingDemoMemReader) Rows() [][]types.Datum { + if r.s == nil { + return nil + } + end := timeNow().Unix() + r.s.windowLock.Lock() + w := r.s.window + if !r.checker.isTimeValid(w.begin.Unix(), end) { + r.s.windowLock.Unlock() + return nil + } + values := w.lru.Values() + evicted := w.evicted + r.s.windowLock.Unlock() + rows := make([][]types.Datum, 0, len(values)) + for _, v := range values { + record := v.(*lockedStmtRecord) + if !r.checker.isDigestValid(record.Digest) { + continue + } + func() { + record.Lock() + defer record.Unlock() + if !r.checker.hasPrivilege(record.AuthUsers) { + return + } + record.Begin = w.begin.Unix() + record.End = end + rows = append(rows, readBillingDemoRowsFromRecord(r, r.columns, record.StmtRecord, r.kind)...) + }() + } + if r.checker.digests == nil { + func() { + evicted.Lock() + defer evicted.Unlock() + if !r.checker.hasPrivilege(evicted.other.AuthUsers) { + return + } + evicted.other.Begin = w.begin.Unix() + evicted.other.End = end + rows = append(rows, readBillingDemoRowsFromRecord(r, r.columns, evicted.other, r.kind)...) + }() + } + return rows +} + // getInstanceAddr implements columnInfo. func (r *MemReader) getInstanceAddr() string { return r.instanceAddr @@ -157,6 +245,16 @@ func (r *MemReader) getTimeLocation() *time.Location { return r.timeLocation } +// getInstanceAddr implements columnInfo. +func (r *ReadBillingDemoMemReader) getInstanceAddr() string { + return r.instanceAddr +} + +// getInstanceAddr implements columnInfo. +func (r *ReadBillingDemoMemReader) getTimeLocation() *time.Location { + return r.timeLocation +} + // HistoryReader is used to read data that has been persisted to files. type HistoryReader struct { ctx context.Context @@ -166,9 +264,12 @@ type HistoryReader struct { instanceAddr string timeLocation *time.Location + columns []*model.ColumnInfo columnFactories []columnFactory checker *stmtChecker files *stmtFiles + readBillingDemo bool + readBillingKind stmtsummarybase.ReadBillingDemoTableKind concurrent int rowsCh <-chan [][]types.Datum @@ -188,6 +289,22 @@ func NewHistoryReader( digests set.StringSet, timeRanges []*StmtTimeRange, concurrent int, +) (*HistoryReader, error) { + return newHistoryReader(ctx, columns, instanceAddr, timeLocation, user, hasProcessPriv, digests, timeRanges, concurrent, false, 0) +} + +func newHistoryReader( + ctx context.Context, + columns []*model.ColumnInfo, + instanceAddr string, + timeLocation *time.Location, + user *auth.UserIdentity, + hasProcessPriv bool, + digests set.StringSet, + timeRanges []*StmtTimeRange, + concurrent int, + readBillingDemo bool, + readBillingKind stmtsummarybase.ReadBillingDemoTableKind, ) (*HistoryReader, error) { files, err := newStmtFiles(ctx, timeRanges) if err != nil { @@ -201,23 +318,30 @@ func NewHistoryReader( errCh := make(chan error, concurrent) ctx, cancel := context.WithCancel(ctx) + var columnFactories []columnFactory + if !readBillingDemo { + columnFactories = makeColumnFactories(columns) + } r := &HistoryReader{ ctx: ctx, cancel: cancel, instanceAddr: instanceAddr, timeLocation: timeLocation, - columnFactories: makeColumnFactories(columns), + columns: columns, + columnFactories: columnFactories, checker: &stmtChecker{ user: user, hasProcessPriv: hasProcessPriv, digests: digests, timeRanges: timeRanges, }, - files: files, - concurrent: concurrent, - rowsCh: rowsCh, - errCh: errCh, + files: files, + readBillingDemo: readBillingDemo, + readBillingKind: readBillingKind, + concurrent: concurrent, + rowsCh: rowsCh, + errCh: errCh, } r.wg.Add(1) @@ -228,6 +352,22 @@ func NewHistoryReader( return r, nil } +// NewReadBillingDemoHistoryReader creates a history reader for persisted read billing demo rows. +func NewReadBillingDemoHistoryReader( + ctx context.Context, + columns []*model.ColumnInfo, + instanceAddr string, + timeLocation *time.Location, + user *auth.UserIdentity, + hasProcessPriv bool, + digests set.StringSet, + timeRanges []*StmtTimeRange, + concurrent int, + kind stmtsummarybase.ReadBillingDemoTableKind, +) (*HistoryReader, error) { + return newHistoryReader(ctx, columns, instanceAddr, timeLocation, user, hasProcessPriv, digests, timeRanges, concurrent, true, kind) +} + // Rows returns rows converted from records in files. Reading and parsing // works asynchronously. If (nil, nil) is returned, it means that the // reading has been completed. @@ -311,7 +451,10 @@ func (r *HistoryReader) scheduleTasks( instanceAddr: r.instanceAddr, timeLocation: r.timeLocation, checker: r.checker, + columns: r.columns, columnFactories: r.columnFactories, + readBillingDemo: r.readBillingDemo, + readBillingKind: r.readBillingKind, } concurrent := r.concurrent @@ -443,6 +586,139 @@ func (c *stmtChecker) needStop(curBegin int64) bool { return stop } +func readBillingDemoRowsFromRecord(info columnInfo, columns []*model.ColumnInfo, record *StmtRecord, kind stmtsummarybase.ReadBillingDemoTableKind) [][]types.Datum { + switch kind { + case stmtsummarybase.ReadBillingDemoTableBaseUnits: + rows := make([][]types.Datum, 0, len(record.ReadBillingDemoBaseUnitAggs)) + for _, entry := range record.ReadBillingDemoBaseUnitAggs { + rows = append(rows, readBillingDemoBaseUnitRow(info, columns, record, entry)) + } + return rows + case stmtsummarybase.ReadBillingDemoTableStatus: + rows := make([][]types.Datum, 0, len(record.ReadBillingDemoStatusAggs)) + for _, entry := range record.ReadBillingDemoStatusAggs { + rows = append(rows, readBillingDemoStatusRow(info, columns, record, entry)) + } + return rows + default: + return nil + } +} + +func readBillingDemoBaseUnitRow(info columnInfo, columns []*model.ColumnInfo, record *StmtRecord, entry stmtsummarybase.ReadBillingDemoBaseUnitAggEntry) []types.Datum { + row := make([]types.Datum, len(columns)) + for i, col := range columns { + row[i] = types.NewDatum(readBillingDemoBaseUnitColumnValue(info, col.Name.O, record, entry)) + } + return row +} + +func readBillingDemoStatusRow(info columnInfo, columns []*model.ColumnInfo, record *StmtRecord, entry stmtsummarybase.ReadBillingDemoStatusAggEntry) []types.Datum { + row := make([]types.Datum, len(columns)) + for i, col := range columns { + row[i] = types.NewDatum(readBillingDemoStatusColumnValue(info, col.Name.O, record, entry)) + } + return row +} + +func readBillingDemoCommonColumnValue(info columnInfo, col string, record *StmtRecord) (any, bool) { + switch col { + case ClusterTableInstanceColumnNameStr: + return info.getInstanceAddr(), true + case SummaryBeginTimeStr: + beginTime := time.Unix(record.Begin, 0) + if beginTime.Location() != info.getTimeLocation() { + beginTime = beginTime.In(info.getTimeLocation()) + } + return types.NewTime(types.FromGoTime(beginTime), mysql.TypeTimestamp, 0), true + case SummaryEndTimeStr: + endTime := time.Unix(record.End, 0) + if endTime.Location() != info.getTimeLocation() { + endTime = endTime.In(info.getTimeLocation()) + } + return types.NewTime(types.FromGoTime(endTime), mysql.TypeTimestamp, 0), true + case StmtTypeStr: + return record.StmtType, true + case SchemaNameStr: + return convertEmptyToNil(record.SchemaName), true + case DigestStr: + return convertEmptyToNil(record.Digest), true + case DigestTextStr: + return record.NormalizedSQL, true + case PlanDigestStr: + return convertEmptyToNil(record.PlanDigest), true + case ResourceGroupName: + return record.ResourceGroupName, true + default: + return nil, false + } +} + +func readBillingDemoBaseUnitColumnValue(info columnInfo, col string, record *StmtRecord, entry stmtsummarybase.ReadBillingDemoBaseUnitAggEntry) any { + if value, ok := readBillingDemoCommonColumnValue(info, col, record); ok { + return value + } + switch col { + case stmtsummarybase.ReadBillingDemoModelVersionStr: + return entry.ModelVersion + case stmtsummarybase.ReadBillingDemoWeightVersionStr: + return entry.WeightVersion + case stmtsummarybase.ReadBillingDemoSiteStr: + return entry.Site + case stmtsummarybase.ReadBillingDemoOpClassStr: + return entry.OpClass + case stmtsummarybase.ReadBillingDemoOperatorKindStr: + return entry.OperatorKind + case stmtsummarybase.ReadBillingDemoUnitStr: + return entry.Unit + case stmtsummarybase.ReadBillingDemoInputSourceStr: + return entry.InputSource + case stmtsummarybase.ReadBillingDemoInputSideStr: + return entry.InputSide + case stmtsummarybase.ReadBillingDemoRowWidthSource: + return entry.RowWidthSource + case stmtsummarybase.ReadBillingDemoValueStr: + return entry.Value + case stmtsummarybase.ReadBillingDemoSampleCountStr: + return entry.SampleCount + case stmtsummarybase.ReadBillingDemoRowWidthSumStr: + return entry.RowWidthSum + case stmtsummarybase.ReadBillingDemoAvgRowWidthStr: + if entry.SampleCount == 0 { + return float64(0) + } + return entry.RowWidthSum / float64(entry.SampleCount) + default: + return nil + } +} + +func readBillingDemoStatusColumnValue(info columnInfo, col string, record *StmtRecord, entry stmtsummarybase.ReadBillingDemoStatusAggEntry) any { + if value, ok := readBillingDemoCommonColumnValue(info, col, record); ok { + return value + } + switch col { + case stmtsummarybase.ReadBillingDemoModelVersionStr: + return entry.ModelVersion + case stmtsummarybase.ReadBillingDemoWeightVersionStr: + return entry.WeightVersion + case stmtsummarybase.ReadBillingDemoSiteStr: + return entry.Site + case stmtsummarybase.ReadBillingDemoOpClassStr: + return entry.OpClass + case stmtsummarybase.ReadBillingDemoOperatorKindStr: + return entry.OperatorKind + case stmtsummarybase.ReadBillingDemoStatusStr: + return entry.Status + case stmtsummarybase.ReadBillingDemoReasonStr: + return entry.Reason + case stmtsummarybase.ReadBillingDemoCountStr: + return entry.Count + default: + return nil + } +} + type stmtTinyRecord struct { Begin int64 `json:"begin"` End int64 `json:"end"` @@ -730,7 +1006,10 @@ type stmtParseWorker struct { instanceAddr string timeLocation *time.Location checker *stmtChecker + columns []*model.ColumnInfo columnFactories []columnFactory + readBillingDemo bool + readBillingKind stmtsummarybase.ReadBillingDemoTableKind } func (w *stmtParseWorker) run( @@ -779,8 +1058,12 @@ func (w *stmtParseWorker) handleLines( continue } - row := w.buildRow(record) - rows = append(rows, row) + if w.readBillingDemo { + rows = append(rows, readBillingDemoRowsFromRecord(w, w.columns, record, w.readBillingKind)...) + } else { + row := w.buildRow(record) + rows = append(rows, row) + } } if len(rows) > 0 { @@ -823,6 +1106,9 @@ func (w *stmtParseWorker) matchConds(record *StmtRecord) bool { if !w.checker.hasPrivilege(record.AuthUsers) { return false } + if !w.readBillingDemo && record.isReadBillingDemoStatusOnly() { + return false + } return true } diff --git a/pkg/util/stmtsummary/v2/reader_test.go b/pkg/util/stmtsummary/v2/reader_test.go index 6f24b1c3422fe..e4aa0c4c18431 100644 --- a/pkg/util/stmtsummary/v2/reader_test.go +++ b/pkg/util/stmtsummary/v2/reader_test.go @@ -19,6 +19,7 @@ import ( "context" "fmt" "os" + "strings" "testing" "time" @@ -28,6 +29,7 @@ import ( "github.com/pingcap/tidb/pkg/types" "github.com/pingcap/tidb/pkg/util" "github.com/pingcap/tidb/pkg/util/set" + stmtsummarybase "github.com/pingcap/tidb/pkg/util/stmtsummary" "github.com/stretchr/testify/require" ) @@ -253,6 +255,114 @@ func TestMemReader(t *testing.T) { require.Len(t, evicted, 3) // begin, end, count } +func TestReadBillingDemoMemReader(t *testing.T) { + timeLocation, err := time.LoadLocation("Asia/Shanghai") + require.NoError(t, err) + + ss := NewStmtSummary4Test(3) + defer ss.Close() + + info := GenerateStmtExecInfo4Test("digest_read_billing") + info.ReadBillingDemoStats = stmtsummarybase.ReadBillingDemoStatementStats{ + ModelVersion: "v1", + WeightVersion: "v1", + Statuses: []stmtsummarybase.ReadBillingDemoStatusSample{{ + ModelVersion: "v1", + WeightVersion: "v1", + Site: "statement", + OpClass: "statement", + OperatorKind: "statement", + Status: "success", + Reason: "none", + }}, + BaseUnits: []stmtsummarybase.ReadBillingDemoBaseUnitSample{{ + ModelVersion: "v1", + WeightVersion: "v1", + Site: "tidb", + OpClass: "projection_eval", + OperatorKind: "projection", + Unit: "fixed_events", + InputSource: "runtime_act_rows", + InputSide: "all", + RowWidthSource: "operator_helper", + Value: 2, + RowWidth: 16, + }}, + Totals: stmtsummarybase.ReadBillingDemoBaseUnitSummary{ + SumReadBillingDemoFixedEvents: 2, + }, + } + ss.Add(info) + + statusOnly := GenerateStmtExecInfo4Test("digest_read_billing_error") + statusOnly.ReadBillingDemoStats = stmtsummarybase.ReadBillingDemoStatementStats{ + ModelVersion: "v1", + WeightVersion: "v1", + Statuses: []stmtsummarybase.ReadBillingDemoStatusSample{{ + ModelVersion: "v1", + WeightVersion: "v1", + Site: "statement", + OpClass: "statement", + OperatorKind: "statement", + Status: "error", + Reason: "statement_error", + }}, + BaseUnits: []stmtsummarybase.ReadBillingDemoBaseUnitSample{{ + ModelVersion: "v1", + WeightVersion: "v1", + Site: "tidb", + OpClass: "projection_eval", + OperatorKind: "projection", + Unit: "fixed_events", + InputSource: "runtime_act_rows", + InputSide: "all", + RowWidthSource: "operator_helper", + Value: 999, + RowWidth: 999, + }}, + Totals: stmtsummarybase.ReadBillingDemoBaseUnitSummary{ + SumReadBillingDemoFixedEvents: 999, + }, + } + ss.AddReadBillingDemoStatusOnly(statusOnly) + + baseUnitCols := []*model.ColumnInfo{ + {Name: ast.NewCIStr(DigestStr)}, + {Name: ast.NewCIStr(stmtsummarybase.ReadBillingDemoSiteStr)}, + {Name: ast.NewCIStr(stmtsummarybase.ReadBillingDemoOpClassStr)}, + {Name: ast.NewCIStr(stmtsummarybase.ReadBillingDemoUnitStr)}, + {Name: ast.NewCIStr(stmtsummarybase.ReadBillingDemoValueStr)}, + {Name: ast.NewCIStr(stmtsummarybase.ReadBillingDemoSampleCountStr)}, + } + baseRows := NewReadBillingDemoMemReader(ss, baseUnitCols, "", timeLocation, nil, false, nil, nil, stmtsummarybase.ReadBillingDemoTableBaseUnits).Rows() + require.Len(t, baseRows, 1) + require.Equal(t, map[string]string{ + "digest_read_billing/tidb/projection_eval/fixed_events": "2 1", + }, readBillingDemoRowsByKey(baseRows, 4)) + + statusCols := []*model.ColumnInfo{ + {Name: ast.NewCIStr(DigestStr)}, + {Name: ast.NewCIStr(stmtsummarybase.ReadBillingDemoStatusStr)}, + {Name: ast.NewCIStr(stmtsummarybase.ReadBillingDemoReasonStr)}, + {Name: ast.NewCIStr(stmtsummarybase.ReadBillingDemoCountStr)}, + } + statusRows := NewReadBillingDemoMemReader(ss, statusCols, "", timeLocation, nil, false, nil, nil, stmtsummarybase.ReadBillingDemoTableStatus).Rows() + require.Len(t, statusRows, 2) + require.Equal(t, map[string]string{ + "digest_read_billing/success/none": "1", + "digest_read_billing_error/error/statement_error": "1", + }, readBillingDemoRowsByKey(statusRows, 3)) + + normalReader := NewMemReader(ss, []*model.ColumnInfo{ + {Name: ast.NewCIStr(DigestStr)}, + {Name: ast.NewCIStr(ExecCountStr)}, + }, "", timeLocation, nil, false, nil, nil) + normalRows := normalReader.Rows() + require.Len(t, normalRows, 1) + require.Equal(t, "digest_read_billing", normalRows[0][0].GetString()) + require.Equal(t, int64(1), normalRows[0][1].GetInt64()) +} + func TestHistoryReader(t *testing.T) { filename1 := "tidb-statements-2022-12-27T16-21-20.245.log" filename2 := "tidb-statements.log" @@ -409,6 +519,61 @@ func TestHistoryReader(t *testing.T) { }() } +func TestReadBillingDemoHistoryReader(t *testing.T) { + filename := "tidb-statements.log" + file, err := os.Create(filename) + require.NoError(t, err) + defer func() { + require.NoError(t, os.Remove(filename)) + }() + _, err = file.WriteString(`{"begin":1672129270,"end":1672129280,"digest":"digest_read_billing","normalized_sql":"select ?","stmt_type":"Select","auth_users":{"user":{}},"read_billing_demo_base_unit_aggs":[{"model_version":"v1","weight_version":"v1","site":"tidb","op_class":"projection_eval","operator_kind":"projection","unit":"fixed_events","input_source":"runtime_act_rows","input_side":"all","row_width_source":"operator_helper","value":2,"sample_count":1,"row_width_sum":16}],"read_billing_demo_status_aggs":[{"model_version":"v1","weight_version":"v1","site":"statement","op_class":"statement","operator_kind":"statement","status":"success","reason":"none","count":1}]}` + "\n") + require.NoError(t, err) + _, err = file.WriteString(`{"begin":1672129270,"end":1672129280,"digest":"digest_read_billing_error","normalized_sql":"select ?","stmt_type":"Select","auth_users":{"user":{}},"read_billing_demo_status_aggs":[{"model_version":"v1","weight_version":"v1","site":"statement","op_class":"statement","operator_kind":"statement","status":"error","reason":"statement_error","count":1}]}` + "\n") + require.NoError(t, err) + require.NoError(t, file.Close()) + + timeLocation, err := time.LoadLocation("Asia/Shanghai") + require.NoError(t, err) + baseUnitCols := []*model.ColumnInfo{ + {Name: ast.NewCIStr(DigestStr)}, + {Name: ast.NewCIStr(stmtsummarybase.ReadBillingDemoSiteStr)}, + {Name: ast.NewCIStr(stmtsummarybase.ReadBillingDemoOpClassStr)}, + {Name: ast.NewCIStr(stmtsummarybase.ReadBillingDemoUnitStr)}, + {Name: ast.NewCIStr(stmtsummarybase.ReadBillingDemoValueStr)}, + } + baseReader, err := NewReadBillingDemoHistoryReader(context.Background(), baseUnitCols, "", timeLocation, nil, false, nil, nil, 2, stmtsummarybase.ReadBillingDemoTableBaseUnits) + require.NoError(t, err) + baseRows := readAllRows(t, baseReader) + require.NoError(t, baseReader.Close()) + require.Equal(t, map[string]string{ + "digest_read_billing/tidb/projection_eval/fixed_events": "2", + }, readBillingDemoRowsByKey(baseRows, 4)) + + statusCols := []*model.ColumnInfo{ + {Name: ast.NewCIStr(DigestStr)}, + {Name: ast.NewCIStr(stmtsummarybase.ReadBillingDemoStatusStr)}, + {Name: ast.NewCIStr(stmtsummarybase.ReadBillingDemoReasonStr)}, + {Name: ast.NewCIStr(stmtsummarybase.ReadBillingDemoCountStr)}, + } + statusReader, err := NewReadBillingDemoHistoryReader(context.Background(), statusCols, "", timeLocation, nil, false, nil, nil, 2, stmtsummarybase.ReadBillingDemoTableStatus) + require.NoError(t, err) + statusRows := readAllRows(t, statusReader) + require.NoError(t, statusReader.Close()) + require.Equal(t, map[string]string{ + "digest_read_billing/success/none": "1", + "digest_read_billing_error/error/statement_error": "1", + }, readBillingDemoRowsByKey(statusRows, 3)) + + normalReader, err := NewHistoryReader(context.Background(), []*model.ColumnInfo{ + {Name: ast.NewCIStr(DigestStr)}, + {Name: ast.NewCIStr(ExecCountStr)}, + }, "", timeLocation, nil, false, nil, nil, 2) + require.NoError(t, err) + normalRows := readAllRows(t, normalReader) + require.NoError(t, normalReader.Close()) + require.Empty(t, normalRows) +} + func TestHistoryReaderInvalidLine(t *testing.T) { filename := "tidb-statements.log" @@ -458,3 +623,19 @@ func readAllRows(t *testing.T, reader *HistoryReader) [][]types.Datum { } return results } + +func readBillingDemoRowsByKey(rows [][]types.Datum, keyColumns int) map[string]string { + result := make(map[string]string, len(rows)) + for _, row := range rows { + keyParts := make([]string, 0, keyColumns) + for i := 0; i < keyColumns; i++ { + keyParts = append(keyParts, fmt.Sprintf("%v", row[i].GetValue())) + } + valueParts := make([]string, 0, len(row)-keyColumns) + for i := keyColumns; i < len(row); i++ { + valueParts = append(valueParts, fmt.Sprintf("%v", row[i].GetValue())) + } + result[strings.Join(keyParts, "/")] = strings.Join(valueParts, " ") + } + return result +} diff --git a/pkg/util/stmtsummary/v2/record.go b/pkg/util/stmtsummary/v2/record.go index fdd9a2de1d471..531b2bb5e5152 100644 --- a/pkg/util/stmtsummary/v2/record.go +++ b/pkg/util/stmtsummary/v2/record.go @@ -155,6 +155,9 @@ type StmtRecord struct { // request units(RU) ResourceGroupName string `json:"resource_group_name"` stmtsummary.StmtRUSummary + stmtsummary.ReadBillingDemoBaseUnitSummary + ReadBillingDemoBaseUnitAggs []stmtsummary.ReadBillingDemoBaseUnitAggEntry `json:"read_billing_demo_base_unit_aggs,omitempty"` + ReadBillingDemoStatusAggs []stmtsummary.ReadBillingDemoStatusAggEntry `json:"read_billing_demo_status_aggs,omitempty"` PlanCacheUnqualifiedCount int64 `json:"plan_cache_unqualified_count"` PlanCacheUnqualifiedLastReason string `json:"plan_cache_unqualified_last_reason"` // the reason why this query is unqualified for the plan cache @@ -237,6 +240,44 @@ func NewStmtRecord(info *stmtsummary.StmtExecInfo) *StmtRecord { } } +// NewReadBillingDemoStatusOnlyRecord creates a minimal record for status-only +// read billing demo rows produced before the normal statement finish path. +func NewReadBillingDemoStatusOnlyRecord(info *stmtsummary.StmtExecInfo) *StmtRecord { + var ok bool + info, ok = stmtsummary.ReadBillingDemoStatusOnlyExecInfo(info) + if !ok { + return &StmtRecord{ + AuthUsers: make(map[string]struct{}), + BackoffTypes: make(map[string]int), + MinLatency: time.Duration(math.MaxInt64), + MinResultRows: math.MaxInt64, + } + } + startTime := info.StartTime + if startTime.IsZero() { + startTime = time.Now() + } + record := &StmtRecord{ + SchemaName: info.SchemaName, + Digest: info.Digest, + PlanDigest: info.PlanDigest, + NormalizedSQL: info.NormalizedSQL, + IsInternal: info.IsInternal, + AuthUsers: make(map[string]struct{}), + BackoffTypes: make(map[string]int), + MinLatency: time.Duration(math.MaxInt64), + MinResultRows: math.MaxInt64, + FirstSeen: startTime, + LastSeen: startTime, + ResourceGroupName: info.ResourceGroupName, + } + if info.StmtCtx != nil { + record.StmtType = info.StmtCtx.StmtType + } + record.AddReadBillingDemoStatusOnly(info) + return record +} + // Add adds the statistics of StmtExecInfo to StmtRecord. func (r *StmtRecord) Add(info *stmtsummary.StmtExecInfo) { r.IsInternal = r.IsInternal && info.IsInternal @@ -444,11 +485,59 @@ func (r *StmtRecord) Add(info *stmtsummary.StmtExecInfo) { r.StmtNetworkTrafficSummary.Add(&tikvExecDetails) // RU r.StmtRUSummary.Add(info.RUDetail, info.TotalRUV2) + if !info.ReadBillingDemoStats.IsEmpty() { + var acceptedSummary stmtsummary.ReadBillingDemoBaseUnitSummary + r.ReadBillingDemoBaseUnitAggs, r.ReadBillingDemoStatusAggs, acceptedSummary = stmtsummary.AddReadBillingDemoStatementStatsToEntries( + r.ReadBillingDemoBaseUnitAggs, + r.ReadBillingDemoStatusAggs, + &info.ReadBillingDemoStats, + ) + r.ReadBillingDemoBaseUnitSummary.Add(&acceptedSummary) + } else { + r.ReadBillingDemoBaseUnitSummary.Add(&info.ReadBillingDemoBaseUnits) + } r.StorageKV = info.StmtCtx.IsTiKV.Load() r.StorageMPP = info.StmtCtx.IsTiFlash.Load() } +// AddReadBillingDemoStatusOnly adds only read billing demo status/base-unit +// aggregates without changing ordinary statement summary counters. +func (r *StmtRecord) AddReadBillingDemoStatusOnly(info *stmtsummary.StmtExecInfo) { + var ok bool + info, ok = stmtsummary.ReadBillingDemoStatusOnlyExecInfo(info) + if !ok { + return + } + if len(info.User) > 0 { + if r.AuthUsers == nil { + r.AuthUsers = make(map[string]struct{}) + } + r.AuthUsers[info.User] = struct{}{} + } + var acceptedSummary stmtsummary.ReadBillingDemoBaseUnitSummary + r.ReadBillingDemoBaseUnitAggs, r.ReadBillingDemoStatusAggs, acceptedSummary = stmtsummary.AddReadBillingDemoStatementStatsToEntries( + r.ReadBillingDemoBaseUnitAggs, + r.ReadBillingDemoStatusAggs, + &info.ReadBillingDemoStats, + ) + r.ReadBillingDemoBaseUnitSummary.Add(&acceptedSummary) +} + +func (r *StmtRecord) isReadBillingDemoStatusOnly() bool { + return r != nil && r.ExecCount == 0 && r.hasReadBillingDemoData() +} + +func (r *StmtRecord) hasReadBillingDemoData() bool { + if r == nil { + return false + } + return len(r.ReadBillingDemoBaseUnitAggs) > 0 || len(r.ReadBillingDemoStatusAggs) > 0 || + r.SumReadBillingDemoFixedEvents != 0 || + r.SumReadBillingDemoInputRows != 0 || + r.SumReadBillingDemoInputBytes != 0 +} + // Merge merges the statistics of another StmtRecord to this StmtRecord. func (r *StmtRecord) Merge(other *StmtRecord) { // User @@ -608,6 +697,18 @@ func (r *StmtRecord) Merge(other *StmtRecord) { r.SumTikvCPU += other.SumTikvCPU r.SumErrors += other.SumErrors r.StmtRUSummary.Merge(&other.StmtRUSummary) + var acceptedSummary stmtsummary.ReadBillingDemoBaseUnitSummary + r.ReadBillingDemoBaseUnitAggs, r.ReadBillingDemoStatusAggs, acceptedSummary = stmtsummary.MergeReadBillingDemoEntrySlices( + r.ReadBillingDemoBaseUnitAggs, + r.ReadBillingDemoStatusAggs, + other.ReadBillingDemoBaseUnitAggs, + other.ReadBillingDemoStatusAggs, + ) + if len(other.ReadBillingDemoBaseUnitAggs) == 0 && len(other.ReadBillingDemoStatusAggs) == 0 { + r.ReadBillingDemoBaseUnitSummary.Merge(&other.ReadBillingDemoBaseUnitSummary) + } else { + r.ReadBillingDemoBaseUnitSummary.Add(&acceptedSummary) + } } // Truncate SQL to maxSQLLength. @@ -712,10 +813,15 @@ func GenerateStmtExecInfo4Test(digest string) *stmtsummary.StmtExecInfo { ResourceGroupName: "rg1", RUDetail: util.NewRUDetailsWith(1.2, 3.4, 2*time.Millisecond), TotalRUV2: 12345, - TiKVExecDetails: &util.ExecDetails{}, - CPUUsages: ppcpuusage.CPUUsages{TidbCPUTime: time.Duration(20), TikvCPUTime: time.Duration(10000)}, - LazyInfo: &mockLazyInfo{}, - MemArbitration: 22222, + ReadBillingDemoBaseUnits: stmtsummary.ReadBillingDemoBaseUnitSummary{ + SumReadBillingDemoFixedEvents: 2, + SumReadBillingDemoInputRows: 100, + SumReadBillingDemoInputBytes: 2048, + }, + TiKVExecDetails: &util.ExecDetails{}, + CPUUsages: ppcpuusage.CPUUsages{TidbCPUTime: time.Duration(20), TikvCPUTime: time.Duration(10000)}, + LazyInfo: &mockLazyInfo{}, + MemArbitration: 22222, } stmtExecInfo.StmtCtx.AddAffectedRows(10000) return stmtExecInfo diff --git a/pkg/util/stmtsummary/v2/record_test.go b/pkg/util/stmtsummary/v2/record_test.go index d35d3807a970f..260004a292a11 100644 --- a/pkg/util/stmtsummary/v2/record_test.go +++ b/pkg/util/stmtsummary/v2/record_test.go @@ -16,9 +16,11 @@ package stmtsummary import ( "encoding/json" + "fmt" "testing" "github.com/pingcap/tidb/pkg/config" + stmtsummarybase "github.com/pingcap/tidb/pkg/util/stmtsummary" "github.com/stretchr/testify/require" ) @@ -67,6 +69,9 @@ func TestStmtRecord(t *testing.T) { require.Equal(t, info.RUDetail.RUWaitDuration(), record1.SumRUWaitDuration) require.Equal(t, info.TotalRUV2, record1.MaxRUV2) require.Equal(t, info.TotalRUV2, record1.SumRUV2) + require.Equal(t, info.ReadBillingDemoBaseUnits.SumReadBillingDemoFixedEvents, record1.SumReadBillingDemoFixedEvents) + require.Equal(t, info.ReadBillingDemoBaseUnits.SumReadBillingDemoInputRows, record1.SumReadBillingDemoInputRows) + require.Equal(t, info.ReadBillingDemoBaseUnits.SumReadBillingDemoInputBytes, record1.SumReadBillingDemoInputBytes) require.Equal(t, info.CPUUsages.TidbCPUTime, record1.SumTidbCPU) require.Equal(t, info.CPUUsages.TikvCPUTime, record1.SumTikvCPU) @@ -83,6 +88,9 @@ func TestStmtRecord(t *testing.T) { require.Equal(t, info.RUDetail.WRU()*2, record2.SumWRU) require.Equal(t, info.RUDetail.RUWaitDuration()*2, record2.SumRUWaitDuration) require.Equal(t, info.TotalRUV2*2, record2.SumRUV2) + require.Equal(t, info.ReadBillingDemoBaseUnits.SumReadBillingDemoFixedEvents*2, record2.SumReadBillingDemoFixedEvents) + require.Equal(t, info.ReadBillingDemoBaseUnits.SumReadBillingDemoInputRows*2, record2.SumReadBillingDemoInputRows) + require.Equal(t, info.ReadBillingDemoBaseUnits.SumReadBillingDemoInputBytes*2, record2.SumReadBillingDemoInputBytes) require.Equal(t, info.CPUUsages.TidbCPUTime*2, record2.SumTidbCPU) require.Equal(t, info.CPUUsages.TikvCPUTime*2, record2.SumTikvCPU) @@ -103,6 +111,7 @@ func TestStmtRecord(t *testing.T) { require.NoError(t, json.Unmarshal(b, &items)) require.Equal(t, map[string]any{"stmt_meta_a": "value_a"}, items["additional_fields"]) require.Equal(t, record2.Digest, items["digest"]) + require.Equal(t, record2.SumReadBillingDemoInputBytes, items["sum_read_billing_demo_input_bytes"]) b, err = marshalEvictedStmtRecord(record2) require.NoError(t, err) @@ -112,3 +121,161 @@ func TestStmtRecord(t *testing.T) { require.Equal(t, true, items["evicted"]) require.Equal(t, record2.Digest, items["digest"]) } + +func TestStmtRecordReadBillingDemoStructuredStats(t *testing.T) { + info := GenerateStmtExecInfo4Test("digest_read_billing") + info.ReadBillingDemoStats = stmtsummarybase.ReadBillingDemoStatementStats{ + ModelVersion: "v1", + WeightVersion: "v1", + Statuses: []stmtsummarybase.ReadBillingDemoStatusSample{{ + ModelVersion: "v1", + WeightVersion: "v1", + Site: "statement", + OpClass: "statement", + OperatorKind: "statement", + Status: "success", + Reason: "none", + }}, + BaseUnits: []stmtsummarybase.ReadBillingDemoBaseUnitSample{ + { + ModelVersion: "v1", + WeightVersion: "v1", + Site: "tidb", + OpClass: "projection_eval", + OperatorKind: "projection", + Unit: "fixed_events", + InputSource: "runtime_act_rows", + InputSide: "all", + RowWidthSource: "operator_helper", + Value: 2, + RowWidth: 16, + }, + { + ModelVersion: "v1", + WeightVersion: "v1", + Site: "tidb", + OpClass: "projection_eval", + OperatorKind: "projection", + Unit: "fixed_events", + InputSource: "runtime_act_rows", + InputSide: "all", + RowWidthSource: "operator_helper", + Value: 4, + RowWidth: 24, + }, + }, + Totals: stmtsummarybase.ReadBillingDemoBaseUnitSummary{ + SumReadBillingDemoFixedEvents: 6, + }, + } + + record := NewStmtRecord(info) + record.Add(info) + require.Equal(t, int64(1), record.ExecCount) + require.Equal(t, 6.0, record.SumReadBillingDemoFixedEvents) + require.Len(t, record.ReadBillingDemoBaseUnitAggs, 1) + require.Equal(t, 6.0, record.ReadBillingDemoBaseUnitAggs[0].Value) + require.Equal(t, uint64(2), record.ReadBillingDemoBaseUnitAggs[0].SampleCount) + require.Equal(t, 40.0, record.ReadBillingDemoBaseUnitAggs[0].RowWidthSum) + require.Len(t, record.ReadBillingDemoStatusAggs, 1) + + b, err := marshalStmtRecord(record) + require.NoError(t, err) + var restored StmtRecord + require.NoError(t, json.Unmarshal(b, &restored)) + require.Equal(t, record.ReadBillingDemoBaseUnitAggs, restored.ReadBillingDemoBaseUnitAggs) + require.Equal(t, record.ReadBillingDemoStatusAggs, restored.ReadBillingDemoStatusAggs) + + statusOnly := GenerateStmtExecInfo4Test("digest_read_billing_error") + statusOnly.ReadBillingDemoStats = stmtsummarybase.ReadBillingDemoStatementStats{ + ModelVersion: "v1", + WeightVersion: "v1", + Statuses: []stmtsummarybase.ReadBillingDemoStatusSample{{ + ModelVersion: "v1", + WeightVersion: "v1", + Site: "statement", + OpClass: "statement", + OperatorKind: "statement", + Status: "error", + Reason: "statement_error", + }}, + BaseUnits: []stmtsummarybase.ReadBillingDemoBaseUnitSample{{ + ModelVersion: "v1", + WeightVersion: "v1", + Site: "tidb", + OpClass: "projection_eval", + OperatorKind: "projection", + Unit: "fixed_events", + InputSource: "runtime_act_rows", + InputSide: "all", + RowWidthSource: "operator_helper", + Value: 999, + RowWidth: 999, + }}, + Totals: stmtsummarybase.ReadBillingDemoBaseUnitSummary{ + SumReadBillingDemoFixedEvents: 999, + }, + } + statusOnlyRecord := NewReadBillingDemoStatusOnlyRecord(statusOnly) + require.Zero(t, statusOnlyRecord.ExecCount) + require.True(t, statusOnlyRecord.isReadBillingDemoStatusOnly()) + require.Len(t, statusOnlyRecord.ReadBillingDemoStatusAggs, 1) + require.Empty(t, statusOnlyRecord.ReadBillingDemoBaseUnitAggs) + require.Zero(t, statusOnlyRecord.SumReadBillingDemoFixedEvents) + require.Contains(t, statusOnlyRecord.AuthUsers, "user") +} + +func TestStmtRecordMergeReadBillingDemoReservedStatusBypassesStatusCap(t *testing.T) { + record := NewStmtRecord(GenerateStmtExecInfo4Test("digest_read_billing_dst")) + for i := 0; i < stmtsummarybase.MaxReadBillingDemoStatusKeysPerRecord; i++ { + record.ReadBillingDemoStatusAggs = append(record.ReadBillingDemoStatusAggs, stmtsummarybase.ReadBillingDemoStatusAggEntry{ + ModelVersion: "v1", + WeightVersion: "v1", + Site: "tidb", + OpClass: fmt.Sprintf("op_%03d", i), + OperatorKind: "projection", + Status: "unsupported", + Reason: "unsupported_operator", + Count: 1, + }) + } + + other := NewStmtRecord(GenerateStmtExecInfo4Test("digest_read_billing_src")) + other.ReadBillingDemoStatusAggs = []stmtsummarybase.ReadBillingDemoStatusAggEntry{ + { + ModelVersion: "v1", + WeightVersion: "v1", + Site: "statement", + OpClass: "statement", + OperatorKind: "statement", + Status: "unknown_input", + Reason: "aggregation_overflow", + Count: 7, + }, + { + ModelVersion: "v1", + WeightVersion: "v1", + Site: "statement", + OpClass: "statement", + OperatorKind: "statement", + Status: "unknown_input", + Reason: "status_aggregation_overflow", + Count: 11, + }, + } + + record.Merge(other) + require.Equal(t, uint64(7), requireReadBillingDemoStatusReason(t, record.ReadBillingDemoStatusAggs, "aggregation_overflow").Count) + require.Equal(t, uint64(11), requireReadBillingDemoStatusReason(t, record.ReadBillingDemoStatusAggs, "status_aggregation_overflow").Count) +} + +func requireReadBillingDemoStatusReason(t *testing.T, entries []stmtsummarybase.ReadBillingDemoStatusAggEntry, reason string) stmtsummarybase.ReadBillingDemoStatusAggEntry { + t.Helper() + for _, entry := range entries { + if entry.Reason == reason { + return entry + } + } + require.Failf(t, "missing read billing status reason", "reason=%s entries=%v", reason, entries) + return stmtsummarybase.ReadBillingDemoStatusAggEntry{} +} diff --git a/pkg/util/stmtsummary/v2/stmtsummary.go b/pkg/util/stmtsummary/v2/stmtsummary.go index 2eab73de1ca2f..b68e44c866d39 100644 --- a/pkg/util/stmtsummary/v2/stmtsummary.go +++ b/pkg/util/stmtsummary/v2/stmtsummary.go @@ -348,6 +348,46 @@ func (s *StmtSummary) Add(info *stmtsummary.StmtExecInfo) { } } +// AddReadBillingDemoStatusOnly adds read billing demo status-only data to the +// current statistics window without changing ordinary statement counters. +func (s *StmtSummary) AddReadBillingDemoStatusOnly(info *stmtsummary.StmtExecInfo) { + var ok bool + info, ok = stmtsummary.ReadBillingDemoStatusOnlyExecInfo(info) + if s.closed.Load() || !ok { + return + } + + k := stmtsummary.StmtDigestKeyPool.Get().(*stmtsummary.StmtDigestKey) + + s.windowLock.Lock() + if s.closed.Load() { + s.windowLock.Unlock() + stmtsummary.StmtDigestKeyPool.Put(k) + return + } + userForKey := "" + if s.optGroupByUser.Load() { + userForKey = info.User + } + k.Init(info.SchemaName, info.Digest, info.PrevSQLDigest, info.PlanDigest, info.ResourceGroupName, userForKey) + var record *lockedStmtRecord + v, exist := s.window.lru.Get(k) + if exist { + record = v.(*lockedStmtRecord) + } else { + record = &lockedStmtRecord{StmtRecord: NewReadBillingDemoStatusOnlyRecord(info)} + s.window.lru.Put(k, record) + } + s.windowLock.Unlock() + + if exist { + record.Lock() + record.AddReadBillingDemoStatusOnly(info) + record.Unlock() + stmtsummary.StmtDigestKeyPool.Put(k) + } +} + // Evicted returns the number of statements evicted for the current // time window. The returned type is one row consisting of three // columns: [BEGIN_TIME, END_TIME, EVICTED_COUNT]. @@ -743,6 +783,24 @@ func Add(stmtExecInfo *stmtsummary.StmtExecInfo) { } } +// AddReadBillingDemoStatusOnly records read billing demo early-error statuses. +func AddReadBillingDemoStatusOnly(stmtExecInfo *stmtsummary.StmtExecInfo) { + var ok bool + stmtExecInfo, ok = stmtsummary.ReadBillingDemoStatusOnlyExecInfo(stmtExecInfo) + if !ok { + return + } + if config.GetGlobalConfig().Instance.StmtSummaryEnablePersistent { + if GlobalStmtSummary == nil || !GlobalStmtSummary.Enabled() || + (stmtExecInfo.IsInternal && !GlobalStmtSummary.EnableInternalQuery()) { + return + } + GlobalStmtSummary.AddReadBillingDemoStatusOnly(stmtExecInfo) + return + } + stmtsummary.StmtSummaryByDigestMap.AddReadBillingDemoStatusOnly(stmtExecInfo) +} + // Enabled wraps GlobalStmtSummary.Enabled and stmtsummary.StmtSummaryByDigestMap.Enabled. func Enabled() bool { if config.GetGlobalConfig().Instance.StmtSummaryEnablePersistent { diff --git a/pkg/util/stmtsummary/v2/stmtsummary_test.go b/pkg/util/stmtsummary/v2/stmtsummary_test.go index c395045e7b387..9e455aeb16120 100644 --- a/pkg/util/stmtsummary/v2/stmtsummary_test.go +++ b/pkg/util/stmtsummary/v2/stmtsummary_test.go @@ -195,6 +195,52 @@ func TestStmtSummaryPersistEvictedDoesNotPersistLoggedRecordsAsAggregate(t *test require.Equal(t, int64(4), totalExecCount) } +func TestStmtSummaryPersistStatusOnlyEvictedAggregate(t *testing.T) { + var logBuf bytes.Buffer + storage := &stmtLogStorage{ + logger: zap.New(zapcore.NewCore(&stmtLogEncoder{}, zapcore.AddSync(&logBuf), zapcore.InfoLevel)), + } + statusOnlyInfo := func(digest string) *stmtsummary.StmtExecInfo { + info := GenerateStmtExecInfo4Test(digest) + info.ReadBillingDemoStats = stmtsummary.ReadBillingDemoStatementStats{ + ModelVersion: "v1", + WeightVersion: "v1", + Statuses: []stmtsummary.ReadBillingDemoStatusSample{{ + ModelVersion: "v1", + WeightVersion: "v1", + Site: "statement", + OpClass: "statement", + OperatorKind: "statement", + Status: "error", + Reason: "statement_error", + }}, + } + return info + } + + ss := NewStmtSummary4Test(1) + ss.storage = storage + ss.AddReadBillingDemoStatusOnly(statusOnlyInfo("digest_status_1")) + ss.AddReadBillingDemoStatusOnly(statusOnlyInfo("digest_status_2")) // evicts digest_status_1 + ss.Close() + + type loggedRecord struct { + Digest string `json:"digest"` + StatusAggs []stmtsummary.ReadBillingDemoStatusAggEntry `json:"read_billing_demo_status_aggs"` + } + var recordsWithStatus []loggedRecord + for _, line := range strings.Split(strings.TrimSpace(logBuf.String()), "\n") { + var record loggedRecord + require.NoError(t, json.Unmarshal([]byte(line), &record)) + if len(record.StatusAggs) > 0 { + recordsWithStatus = append(recordsWithStatus, record) + } + } + require.Len(t, recordsWithStatus, 2) + require.Contains(t, []string{recordsWithStatus[0].Digest, recordsWithStatus[1].Digest}, "") + require.Contains(t, []string{recordsWithStatus[0].Digest, recordsWithStatus[1].Digest}, "digest_status_2") +} + func TestStmtSummaryGroupByUser(t *testing.T) { ss := NewStmtSummary4Test(100) defer ss.Close()