Prometheus: update resource translation

[dashpole]

Part of #4753

Fixes #4577

Related to open-telemetry/opentelemetry-collector-contrib#45982

Issues with the current state

• The receiver can get both job (from scrape config) and service.name from target info. Currently, we let one win, but ideally we would keep both.
• Users that push OTLP to prometheus still expect to see service.name and service.instance.id, and are confused when it is missing. We should consider keeping those and having a job + instance. We can add configuration to disable it, if necessary.

Changes

There are a bunch of changes to the formatting and wording included in this PR. If needed, I can split those out.

Add prometheus.job and prometheus.instance as labels based on job and instance from receivers. In the receiver, we can give users the option not to default these to service.name and service.instance.id. This solves the issue we currently have where service.name can be present in target_info, and conflict with the job and instance from the prometheus target. This would allow us to preserve both in those cases. On the exporter side, we translate them back to job and instance, and can also preserve the original service.name and service.instance.id separately.

Alternatives

• Use job and instance instead of prometheus.job or prometheus.instance.
• Default to not producing the new prometheus.* attributes. This means less change for users, but more issues with the current state where job or service.name has to be dropped.

@open-telemetry/prometheus-interoperability @aknuds1 @kln21002 @cyrille-leclerc

[ArthurSens]

Hmmmm, to make quicker progress here, would it make sense to focus on the Prometheus -> OTLP transformation and leave OTLP -> Prometheus in a separate PR?

I feel like OTLP -> Prometheus is a lot more complicated, and will require deeper discussions with other parts of OTel, like Infrastructure semantic conventions, OBI, etc 🤔

[dashpole]

My main goal is to make sure that everything still round-trips properly. If I make changes to one direction, I usually have to make the inverse change to the other direction to ensure that. I can remove some of the editorial changes to the OTLP -> Prometheus section if that helps. But I would like aggregated exporters to accept prometheus.job and prometheus.instance if we add those to the receiver.

[aknuds1]

Please see comments :)

[github-actions]

This PR was marked stale. It will be closed in 14 days without additional activity.

[dashpole]

PoC in the collector: open-telemetry/opentelemetry-collector-contrib@main...dashpole:opentelemetry-collector-contrib:promjob_poc

[jmacd]

I wonder if job has the form of namespace/name whether it should be used to set service.namespace accordingly. I was trying to square this with the text on lines 640-642, which would insert the service.namespace, and then if there was another level of federation the service.namespace value might disappear.

[dashpole]

job would be formed that way if you are using a prometheus or PRW collector exporter (since SDKs don't populate job/instance themselves), and then need to translate it back to OTLP at a future point. Generally, that isn't a thing that happens much in practice, since it is much easier to just send OTLP than to round-trip through PRW or the Prometheus format. So it is something we could do, but probably has little value in practice.

Overall, I'm trying to deemphasize job == service.name in this PR to allow them to be separate things conceptually.

[jmacd]

I left a very minor question.

[ArthurSens]

I did a pass through the PR, and so far, I have just one comment.

I feel like I haven't fully grasped the potential consequences of this change, so I'm taking some extra time to reflect, I hope that's ok :)