feat(rta): extend Real-Time Analytics to MySQL#5509
Draft
theTibi wants to merge 8 commits into
Draft
Conversation
theTibi
requested review from
4nte,
JiriCtvrtka,
matejkubinec and
mattiasimonato
and removed request for
a team
Contributor
|
|
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #5509 +/- ##
==========================================
- Coverage 43.50% 43.22% -0.29%
==========================================
Files 413 417 +4
Lines 42953 43450 +497
==========================================
+ Hits 18687 18780 +93
- Misses 22392 22795 +403
- Partials 1874 1875 +1
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
1 task
maxkondr
reviewed
Jun 19, 2026
Comment on lines
+34
to
+39
| // MySQL RTA shipped in 3.8.0 — an agent in [3.7.0, 3.8.0) supports MongoDB RTA but | ||
| // would not understand the MySQL builtin, so it must be reported as unsupported. | ||
| assert.False(t, isRtaFeatureSupported("3.7.0", models.MySQLServiceType)) | ||
| assert.False(t, isRtaFeatureSupported("3.7.99", models.MySQLServiceType)) | ||
| assert.True(t, isRtaFeatureSupported("3.8.0", models.MySQLServiceType)) | ||
| assert.True(t, isRtaFeatureSupported("3.9.0", models.MySQLServiceType)) |
Contributor
There was a problem hiding this comment.
Suggested change
| // MySQL RTA shipped in 3.8.0 — an agent in [3.7.0, 3.8.0) supports MongoDB RTA but | |
| // would not understand the MySQL builtin, so it must be reported as unsupported. | |
| assert.False(t, isRtaFeatureSupported("3.7.0", models.MySQLServiceType)) | |
| assert.False(t, isRtaFeatureSupported("3.7.99", models.MySQLServiceType)) | |
| assert.True(t, isRtaFeatureSupported("3.8.0", models.MySQLServiceType)) | |
| assert.True(t, isRtaFeatureSupported("3.9.0", models.MySQLServiceType)) | |
| // MySQL RTA shipped in 3.9.0 — an agent in [3.7.0, 3.8.0) supports MongoDB RTA but | |
| // would not understand the MySQL builtin, so it must be reported as unsupported. | |
| assert.False(t, isRtaFeatureSupported("3.7.0", models.MySQLServiceType)) | |
| assert.False(t, isRtaFeatureSupported("3.7.99", models.MySQLServiceType)) | |
| assert.False(t, isRtaFeatureSupported("3.8.0", models.MySQLServiceType)) | |
| assert.True(t, isRtaFeatureSupported("3.9.0", models.MySQLServiceType)) |
| } | ||
|
|
||
| return versionParsed.IsFeatureSupported(version.MongoDBRtaAgentSupportVersion) | ||
| return versionParsed.IsFeatureSupported(rtaMinAgentVersion(serviceType)) |
Contributor
There was a problem hiding this comment.
what will be in result if someone calls this func with pmmAgentVersion='3.9.0', servicetype=Valkey?
| defer cancel() | ||
|
|
||
| rows, err := m.db.QueryContext(queryCtx, currentQueriesSQL) | ||
| if err != nil { |
Contributor
There was a problem hiding this comment.
as I remember it is needed to separate errors:
- NoRows -> result list is just empty - no need to produce error
- other errors - return error
Contributor
Author
There was a problem hiding this comment.
Good point to double-check. Here QueryContext (not QueryRow) is used, so it never returns sql.ErrNoRows — an empty processlist just yields zero rows.Next() iterations and we return an empty slice with a nil error, which is exactly the "empty list, no error" behaviour. The only way into this if err != nil branch is a real query failure.
What was genuinely off is the message: "not available or permission denied" was a misleading catch-all (it would mislabel e.g. a transient timeout), and it's now redundant since availability/permissions are validated up front by the startup preflight. I've reworded it to a neutral "failed to query sys.x$processlist: %w" and added a comment noting the empty-result semantics (a97f0c8).
Generated by Claude Code
| return nil, "", err | ||
| } | ||
|
|
||
| // RTA agent needs only a single short-lived connection at a time. |
Contributor
There was a problem hiding this comment.
why short-lived? it needs long-lived
Contributor
|
More points missed:
|
Adds MySQL support to Real-Time Analytics (RTA) alongside the existing MongoDB implementation. Running queries are sourced from the sys schema processlist (sys.x$processlist), mirroring the MongoDB currentOp flow. API: - query.proto: new QueryMySQLData payload added to QueryData oneof - realtimeanalytics.proto: ListServicesResponse now returns mysql services - inventory agents.proto: new AGENT_TYPE_RTA_MYSQL_AGENT (20) and RTAMySQLAgent message; wired into List/Get agent responses Agent: - new agent/agents/mysql/realtimeanalytics collector that periodically reads currently running statements from sys.x$processlist and streams them to the server - supervisor wiring for AGENT_TYPE_RTA_MYSQL_AGENT Managed: - RTAMySQLAgentType model + DSN/Files/compatibility/agent-type wiring - realtimeanalytics service: ListServices/StartSession support MySQL, getRTAAgentTypeForServiceType maps MySQL service -> RTA MySQL agent - rtaMySQLAgentConfig built-in agent state; converters + inventory grpc server handle the new agent type UI: - rta types: QueryMySQLData payload, mysql in available services - QueryAndDetails renders MySQL-specific metrics (command, state, program name, rows examined/sent, full scan) and uses SQL highlighting - overview query cell + syntax highlighter gain SQL language support - selection requests MySQL services; disclaimer mentions MySQL
Addresses review feedback on the MySQL RTA: - Raw data now mirrors the MongoDB agent: the collector selects the full sys.x$processlist row (SELECT *) and stores every column in query_raw_json, pretty-printed with json.MarshalIndent. Numeric columns are kept as numbers and SQL NULLs as null. The details view keeps a curated subset. This surfaces execution_engine, lock_latency, cpu_latency, rows_affected, tmp_tables, trx_state/latency, pid, current_memory, etc. - Overview gains a "Hide COMMIT" toolbar toggle that filters bare transaction-control statements (COMMIT/ROLLBACK/BEGIN/START TRANSACTION), which can dominate the list under transactional workloads. Data is still collected; the toggle only affects the view. - Added unit tests for the queryLanguage and isTransactionControl helpers.
Addresses code-review feedback on the MySQL RTA: - Version gate (correctness): MySQL RTA is now gated on a dedicated MySQLRtaAgentSupportVersion (3.8.0), not the MongoDB 3.7.0 version. isRtaFeatureSupported takes the service type so ListServices/StartSession no longer enable MySQL RTA against agents in [3.7.0, 3.8.0) that lack the AGENT_TYPE_RTA_MYSQL_AGENT builtin and would dead-end in the supervisor. Disclaimer updated to "MongoDB (3.7.0+) and MySQL (3.8.0+)". - Minimum-duration floor: the collector now filters statement_latency >= 10ms in SQL, mirroring the MongoDB collector's microsecs_running >= 10_000 and avoiding large buckets of sub-ms statements. (The Hide-COMMIT toggle remains, since durable commits exceed the floor.) - Tests: added Go table tests for coerceValue/mapString/mapInt/mapFloat and buildQueryData (latency math, full_scan, NULL handling, raw JSON), plus a unit test for the per-type version gate. - Nits: fixed shutdown race in the collector (WaitGroup before close), documented coerceValue's numeric-coercion caveat, corrected the stale "MongoDB only" comment and the proto doc (sys.x$processlist), reused the shared queryLanguage() helper in QueryAndDetails, and hardened isTransactionControl (trailing ';', WORK keyword, whitespace).
Drop the 10ms statement_latency filter from the MySQL RTA collector so all
currently-running statements are collected, not only those running >= 10ms.
Idle ("Sleep"/"Daemon") connections, the agent's own connection and rows
without a current statement are still excluded. The Hide-COMMIT view toggle
remains for filtering transaction-control noise.
Before reporting RUNNING, the MySQL RTA agent now verifies the instance can actually serve Real-Time Analytics and fails with AGENT_STATUS_INITIALIZATION_ERROR (surfaced as a session ERROR) otherwise, instead of looping silently with no data: - reject MariaDB (its performance_schema/sys schema differ and have no compatible sys.x$processlist), detected via the shared version.GetMySQLVersion helper; - require performance_schema to be enabled; - require sys.x$processlist to be readable by the monitoring user (catches missing schema and insufficient privileges). A connection failure is now also reported as INITIALIZATION_ERROR. This closes the silent-failure gap noted in review for MariaDB, disabled performance_schema and permission problems.
- Gate MySQL RTA on 3.9.0 (the release that ships the collector) instead of 3.8.0; add V3_9_0 and update the disclaimer and gate test accordingly. - isRtaFeatureSupported now returns false for service types that do not support RTA at all (e.g. Valkey/PostgreSQL) instead of falling back to the MongoDB version; rtaMinAgentVersion returns (version, ok). - Reword the collector's query error: an empty processlist is not an error (QueryContext never returns ErrNoRows), so the catch-all "not available or permission denied" message becomes a neutral "failed to query sys.x$processlist" (availability/permissions are already validated by the startup preflight). - Fix the misleading connection comment: the pool keeps a single long-lived, reused connection (ConnMaxLifetime=0), not a short-lived one.
… agent Brings the MySQL RTA agent to parity with MongoDB: - proto: AddRTAMySQLAgentParams / ChangeRTAMySQLAgentParams and the rta_mysql_agent entries in Add/ChangeAgentRequest/Response oneofs; regenerated Go + swagger spec/clients. - managed: AgentsService.AddRTAMySQLAgent / ChangeRTAMySQLAgent and the inventory gRPC AddAgent/ChangeAgent wiring. - pmm-admin: `inventory add agent rta-mysql-agent` and `inventory change agent rta-mysql-agent`, plus list-agents rendering. - api-tests: TestRTAMySQLAgent (add/get/change, partial update, validation errors); AddAgent test helper handles the new agent type.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Extends Real-Time Analytics (RTA) — previously MongoDB-only — to MySQL. Currently-running queries are collected from the MySQL
sysschema processlist (sys.x$processlist, the machine-readable variant ofsys.processlist), mirroring the existing MongoDBcurrentOpflow.A user can start an RTA session for a MySQL service, see it in the Real-time sessions list, and watch live running queries (with elapsed time, host, and a details pane of MySQL-specific attributes) in the Real-time overview.
Changes
API / proto
query.proto: newQueryMySQLDatapayload added to theQueryDataoneof (command, state, program name, rows examined/sent, full scan, db instance address, database, user).realtimeanalytics.proto:ListServicesResponsenow also returnsmysqlservices.inventory/agents.proto: newAGENT_TYPE_RTA_MYSQL_AGENT(20) andRTAMySQLAgentmessage, wired into List/Get agent responses.Agent
agent/agents/mysql/realtimeanalyticscollector that periodically reads currently-running statements fromsys.x$processlistand streams them to the server.query_raw_json, pretty-printed (numbers stay numbers, SQL NULLs becomenull) — mirroring how the MongoDB agent dumps the wholecurrentOpdocument.Managed
RTAMySQLAgentTypemodel with DSN / TLS-files / compatibility / agent-type wiring.realtimeanalyticsservice:ListServices/StartSessionsupport MySQL;getRTAAgentTypeForServiceTypemaps a MySQL service to the RTA MySQL agent.rtaMySQLAgentConfigbuilt-in agent state; converters and inventory gRPC server handle the new agent type.UI
QueryMySQLDatapayload andmysqlavailable services.QueryAndDetailsrenders MySQL-specific metrics (command, state, program name, rows examined/sent, full scan) and uses SQL syntax highlighting; the overview query cell and syntax highlighter gain SQL support; the Raw-data tab shows the complete, formatted processlist row.Testing
RUNNING, and live queries fromsys.x$processlistflow through to the overview/details/raw views with the MySQL payload populated; the Hide-COMMIT toggle removes transaction-control noise.go vetis clean;versionunit tests pass.tscpasses and the RTA unit tests pass (added MySQL cases toQueryAndDetailsplusqueryLanguage/isTransactionControlunit tests).