Skip to content

Prometheus: update resource translation #4956

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
dashpole wants to merge 6 commits into
base: main
Choose a base branch
from
+68 −56
Open

Prometheus: update resource translation #4956

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
cd2a519
prometheus: update resource section
dashpole Mar 17, 2026
70a1755
changelog
dashpole Mar 17, 2026
669e7da
Update specification/compatibility/prometheus_and_openmetrics.md
dashpole Mar 19, 2026
8b25f40
Merge branch 'main' into prom_stabilize_resource
dashpole Apr 7, 2026
720c3d2
Merge branch 'main' into prom_stabilize_resource
dashpole Apr 10, 2026
7248acd
Merge branch 'main' into prom_stabilize_resource
dashpole Apr 13, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,8 @@ release.

- Stabilize Prometheus Classic Histogram to OTLP Explicit Histogram transformation.
([#4874](https://github.com/open-telemetry/opentelemetry-specification/pull/4874))
- Prometheus `job` and `instance` are now translated to `prometheus.job` and `prometheus.instance`.
([#4956](https://github.com/open-telemetry/opentelemetry-specification/pull/4956))

### SDK Configuration

Expand Down
122 changes: 66 additions & 56 deletions specification/compatibility/prometheus_and_openmetrics.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -266,38 +266,44 @@ Prometheus receiver).

**Status**: [Development](../document-status.md)

When scraping a Prometheus endpoint, resource attributes MUST be added to the
scraped metrics to distinguish them from metrics from other Prometheus
endpoints. In particular, `service.name` and `service.instance.id`, are needed
to ensure Prometheus exporters can disambiguate metrics using
[`job` and `instance` labels](https://prometheus.io/docs/concepts/jobs_instances/)
as [described below](#resource-attributes-1).
Prometheus identifies the origin of metrics by [`job` and `instance` labels](https://prometheus.io/docs/concepts/jobs_instances/).
These labels are added when the metrics are scraped from the Prometheus endpoint,
where the `job` is set based on the scrape configuration, and the `instance` is
set to the target's address. Even though these are added at scrape time, they
are considered identifying for metrics regardless of whether they are scraped or
pushed (using Prometheus Remote Write).

The following attributes MUST be associated with scraped metrics as resource
attributes, and MUST NOT be added as metric attributes:
Prometheus also stores metadata associated with scraped targets in the
[target_info](https://github.com/prometheus/OpenMetrics/blob/v1.0.0/specification/OpenMetrics.md#supporting-target-metadata-in-both-push-based-and-pull-based-systems)
Metric (which belongs to the `target` MetricFamily in OM 1.0).

| OTLP Resource Attribute | Description |
| ----------------------- | ----------- |
| `service.name` | The configured name of the service that the target belongs to |
| `service.instance.id` | A unique identifier of the target. By default, it should be the `<host>:<port>` of the scraped URL |
`job` and `instance` labels SHOULD be stored as `prometheus.job` and
`prometheus.instance` resource attributes, respectively.

If present, the `target` info metric MUST be dropped from the batch of metrics,
and all labels from the `target` info metric MUST be converted to resource
Comment thread
dashpole marked this conversation as resolved.
Outdated
attributes attached to all other metrics which are part of the scrape. By
default, label keys and values MUST NOT be altered (such as replacing `_` with
`.` characters in keys).

The following attributes SHOULD be associated with scraped metrics as resource
attributes, and MUST NOT be added as metric attributes:
If `service.name` or `service.instance.id` are not provided by `target_info`,
they MUST be defaulted to the `job` and `instance` labels, respectively.
Copy link
Copy Markdown
Contributor

@jmacd jmacd Apr 14, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wonder if job has the form of namespace/name whether it should be used to set service.namespace accordingly. I was trying to square this with the text on lines 640-642, which would insert the service.namespace, and then if there was another level of federation the service.namespace value might disappear.

Copy link
Copy Markdown
Contributor Author

@dashpole dashpole Apr 15, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

job would be formed that way if you are using a prometheus or PRW collector exporter (since SDKs don't populate job/instance themselves), and then need to translate it back to OTLP at a future point. Generally, that isn't a thing that happens much in practice, since it is much easier to just send OTLP than to round-trip through PRW or the Prometheus format. So it is something we could do, but probably has little value in practice.

Overall, I'm trying to deemphasize job == service.name in this PR to allow them to be separate things conceptually.

Comment on lines +391 to +392
Copy link
Copy Markdown
Member

@bogdandrutu bogdandrutu Apr 25, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What if in the batch there are metrics with different job/instance pair?


#### Resource Attributes for Scraped Targets

When Prometheus metrics are scraped from a Prometheus endpoint, the following
attributes SHOULD be associated with scraped metrics as resource attributes,
and MUST NOT be added as metric attributes:

| OTLP Resource Attribute | Description |
| ----------------------- | ----------- |
| `server.address` | The `<host>` portion of the target's URL that was scraped |
| `server.port` | The `<port>` portion of the target's URL that was scraped |
| `url.scheme` | `http` or `https` |
Comment on lines 400 to 404
Copy link
Copy Markdown
Member

@bogdandrutu bogdandrutu Apr 25, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we not put job/instance as the service.name/service.instance.id?


In addition to the attributes above, the
[target](https://github.com/prometheus/OpenMetrics/blob/v1.0.0/specification/OpenMetrics.md#supporting-target-metadata-in-both-push-based-and-pull-based-systems)
info metric is used to supply additional resource attributes. If present,
the `target` info metric MUST be dropped from the batch of metrics, and all labels from
the `target` info metric MUST be converted to resource attributes
attached to all other metrics which are part of the scrape. By default, label
keys and values MUST NOT be altered (such as replacing `_` with `.` characters
in keys).
If the target was discovered using service discovery, metadata about the
discovered target SHOULD be added as resource attributes, using OpenTelemetry
semantic conventions where possible.
Comment thread
dashpole marked this conversation as resolved.

## OTLP Metric points to Prometheus

Expand Down Expand Up @@ -494,42 +500,46 @@ Prometheus exemplar unless they would exceed the

**Status**: [Development](../document-status.md)

In Prometheus exporters, an OpenTelemetry Resource SHOULD be converted to
a [`target` info metric](https://github.com/prometheus/OpenMetrics/blob/v1.0.0/specification/OpenMetrics.md#supporting-target-metadata-in-both-push-based-and-pull-based-systems)
OpenTelemetry Resource attributes SHOULD be converted to a
[`target_info` Metric](https://github.com/prometheus/OpenMetrics/blob/v1.0.0/specification/OpenMetrics.md#supporting-target-metadata-in-both-push-based-and-pull-based-systems)
if the resource is not [empty](../resource/sdk.md#the-empty-resource).
The Resource attributes MAY be copied to labels of exported metric families
if required by the exporter configuration, or MUST be dropped. The `target`
info metric MUST be an info-typed
metric whose labels MUST include the resource attributes, and MUST NOT include
any other labels.

In the Collector's Prometheus exporters, it is
possible for metrics from multiple targets to be sent together, so targets must
be disambiguated from one another. However, the Prometheus exposition format
and [remote-write](https://github.com/Prometheus/Prometheus/blob/main/prompb/remote.proto)
formats do not include a notion of resource, and expect metric labels to
distinguish scraped targets. By convention, [`job` and `instance`](https://prometheus.io/docs/concepts/jobs_instances/)
labels distinguish targets and are expected to be present on metrics exposed on
a Prometheus pull exporter (a ["federated"](https://prometheus.io/docs/prometheus/latest/federation/)
Prometheus endpoint) or pushed via Prometheus remote-write. In OpenTelemetry
semantic conventions, the `service.name`, `service.namespace`, and
`service.instance.id` triplet is
[required to be unique](https://github.com/open-telemetry/semantic-conventions/blob/main/docs/resource/README.md#service),
which makes them good candidates to use to construct `job` and `instance`. In
the collector Prometheus exporters, the `service.name` and `service.namespace`
attributes MUST be combined as `<service.namespace>/<service.name>`, or
`<service.name>` if namespace is empty, to form the `job` metric label. The
`service.instance.id` attribute, if present, MUST be converted to the
The Resource attributes MUST NOT be copied to labels of exported metric families
by default. The `target_info` Metric MUST be an info-typed metric whose labels
MUST include the resource attributes, and MUST NOT include any other labels.

If info-typed metric families are not yet supported by the language Prometheus
client library, a gauge-typed metric family named `target_info` with a constant
value of 1 MUST be used instead.

To convert OTLP resource attributes to Prometheus labels, string Attribute
values are converted directly to labels, and non-string Attribute values MUST be
converted to string attributes following the [attribute
specification](../common/README.md#attribute).

#### Aggregated Exporters

Unlike application-level Prometheus exporters, exporters which expose aggregated
metrics from multiple applications are expected to include
[`job` and `instance`](https://prometheus.io/docs/concepts/jobs_instances/)
labels to distinguish the source of the metrics. This is the case for both
["federated"](https://prometheus.io/docs/prometheus/latest/federation/) pull
metric endpoints, such as the Collector's Prometheus exporter, and Prometheus
push metric exporters, such as the Collector's
[Prometheus Remote Write](https://prometheus.io/docs/specs/prw/remote_write_spec_2_0/)
exporter.

If `prometheus.job` or `prometheus.instance` are present in the OpenTelemetry
resource attributes, they MUST be used as the `job` and `instance` labels,
Comment thread
dashpole marked this conversation as resolved.
respectively, and MUST NOT be included on target_info as `prometheus.job` and
`prometheus.instance`. If `prometheus.job` is not present, the `service.name`
and `service.namespace` attributes MUST be combined as
`<service.namespace>/<service.name>`, or `<service.name>` if namespace is empty,
to form the `job` metric label. If `prometheus.instance` is not present,
the `service.instance.id` attribute, if present, MUST be converted to the
`instance` label; otherwise, `instance` should be added with an empty value.
Other resource attributes SHOULD be converted to a
[target](https://github.com/prometheus/OpenMetrics/blob/v1.0.0/specification/OpenMetrics.md#supporting-target-metadata-in-both-push-based-and-pull-based-systems)
info metric, or MUST be dropped. The `target` metric is an info-typed metric
whose labels MUST include the resource attributes, and MUST NOT include any
other labels other than `job` and `instance`. There MUST be at most one
`target` info metric exported for each unique combination of `job` and `instance`.

If info-typed metric families are not yet supported by the language Prometheus client library, a gauge-typed metric family named `target_info` with a constant value of 1 MUST be used instead.

To convert OTLP resource attributes to Prometheus labels, string Attribute values are converted directly to labels, and non-string Attribute values MUST be converted to string attributes following the [attribute specification](../common/README.md#attribute).
The resulting `job` and `instance` labels MUST be added to all OpenTelemetry
metrics that associated with the Resource, including the `target_info` metric,
if present.

[metricMetadata]: https://github.com/open-telemetry/opentelemetry-proto/blob/c451441d7b73f702d1647574c730daf7786f188c/opentelemetry/proto/metrics/v1/metrics.proto#L199
Loading