Skip to content

Go: Add opt-in CPU metric, and detailed opt-in attributes#3627

Merged
lmolkova merged 6 commits intoopen-telemetry:mainfrom
dashpole:future-proof-go-metrics
Apr 27, 2026
Merged

Go: Add opt-in CPU metric, and detailed opt-in attributes#3627
lmolkova merged 6 commits intoopen-telemetry:mainfrom
dashpole:future-proof-go-metrics

Conversation

@dashpole
Copy link
Copy Markdown
Contributor

@dashpole dashpole commented Apr 15, 2026

Fixes #3354

Related to open-telemetry/opentelemetry-go-contrib#6321

Also includes a request from golang/go#67120 (comment) for detailed CPU metrics.

Changes

All additions are all opt-in:

  • Adds go.cpu.time, which has go.cpu.state and go.cpu.detailed_state.
  • Adds go.memory.detailed_type to go.memory.usage.

My goals with this are:

  • Provide users with a default set of high-level, stable attributes for CPU and memory metrics.
  • Provide advanced users a way to get more detailed attributes for the same metrics that may change with each go version. Don't break usage of the default set of attributes when this is enabled.
  • Use the tools provided by the metrics API (opt-in metrics coming soon, opt-in attributes).

The approach i've taken is to use two attributes:

  • One default-on attribute: go.cpu.state or go.memory.type, which has a set of stable, well-defined constants that are useful to most users and highly unlikely to change. E.g. "stack" for memory, or "gc" for CPU.
  • One opt-in attribute: go.cpu.detailed_state and go.memory.detailed_type, which will provide all of the categories offered by the go runtime for each version: https://pkg.go.dev/runtime/metrics#hdr-Supported_metrics based on a prefix (cpu/classes/* and memory/classes/* respectively).

Enabling the opt-in attribute increases cardinality, since multiple detailed types/states can apply to each of the high-level types/states.

Alternatives Considered

We could instead have separate metrics for detailed and basic attributes. This is more expensive without much benefit.

We could instead change the values reported to the detailed ones based on user configuration. This would break existing usage of the high-level attributes when a user enables the detailed ones.

Merge requirement checklist

Gemini helped me copy descriptions over from the runtime metrics documentation.

@open-telemetry/go-approvers @jmacd

@github-actions github-actions Bot added enhancement New feature or request area:go labels Apr 15, 2026
@dashpole dashpole force-pushed the future-proof-go-metrics branch 3 times, most recently from 92f13d6 to d04c7e2 Compare April 15, 2026 20:25
@dashpole dashpole marked this pull request as ready for review April 15, 2026 20:26
@dashpole dashpole requested review from a team as code owners April 15, 2026 20:26
Comment thread docs/runtime/go-metrics.md
Comment thread docs/runtime/go-metrics.md Outdated
@dashpole dashpole force-pushed the future-proof-go-metrics branch from d04c7e2 to 9d0a7d5 Compare April 21, 2026 13:36
Comment thread docs/runtime/go-metrics.md Outdated
Comment thread docs/runtime/go-metrics.md Outdated
Comment thread docs/runtime/go-metrics.md
Comment thread docs/runtime/go-metrics.md Outdated
Copy link
Copy Markdown
Contributor

@jmacd jmacd left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you @dashpole this looks great.

@joaopgrassi
Copy link
Copy Markdown
Member

joaopgrassi commented Apr 23, 2026

One thing that may be confusing with the taken approach is that users may expect that go.memory.detailed_type to be a superset of go.memory.type where in fact it's not. So say the split they metric by the detailed dimension thinking they would also see the "stable" values but they won't.

We could instead change the values reported to the detailed ones based on user configuration. This would break existing usage of the high-level attributes when a user enables the detailed ones.

Did you mean change the values reported on the go.memory.type to include the unstable, detailed ones? I don't really follow how it would break existing usage.

@dashpole
Copy link
Copy Markdown
Contributor Author

Did you mean change the values reported on the go.memory.type to include the unstable, detailed ones? I don't really follow how it would break existing usage.

The sum of all attributes must sum to the memory usage of the go process, otherwise it isn't useful. So we can aggregate the detailed, unstable values into higher-leveled ones, but we can't simply omit categories of memory usage without breaking the total. When we aggregate, it would be misleading to re-use the same attribute values to mean different things.

Comment thread .chloggen/go-detailed-cpu-memory.yaml
Comment thread docs/runtime/go-metrics.md
Comment thread model/go/metrics.yaml Outdated
Copy link
Copy Markdown
Member

@joaopgrassi joaopgrassi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like the tables need updating, after that we can merge.

@dashpole dashpole force-pushed the future-proof-go-metrics branch from a7526bf to dae9312 Compare April 24, 2026 20:43
@lmolkova lmolkova added this pull request to the merge queue Apr 27, 2026
Merged via the queue into open-telemetry:main with commit 59a778d Apr 27, 2026
18 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:go enhancement New feature or request

Projects

Archived in project

Development

Successfully merging this pull request may close these issues.

Go Runtime Metrics: include metric to replace process.runtime.go.mem.heap_sys

8 participants