feat: Explicitly define include/exclude attributes for views

[thompson-tomo]

Fixes #

Changes

Explicitly define method to exclude metric attributes which was already mentioned functionality in the attributes property description in the spec. This exclude functionality was already defined in the stable declarative config. The include is added to avoid changing the behaviour of the existing property.

Discussed on slack -> https://cloud-native.slack.com/archives/C01N7PP1THC/p1773667594006789

For non-trivial changes, follow the change proposal process.

• Related issues #
• Related OTEP(s) #
• Links to the prototypes (when adding or changing features)
• CHANGELOG.md file updated for non-trivial changes
  • For trivial changes, include [chore] in the PR title to skip the changelog check
• Spec compliance matrix updated if necessary

[cijothomas]

don't need this in the API/advisory. if something should not be used, then instrumentation can not provide it at all.

[thompson-tomo]

Wouldn't opt-in/deprecated attributes reside in this advisory? That way if stream config doesn't specify included/excluded attributes the advisory can be used to exclude them with 0 zero config.

[cijothomas]

Opt-in attributes are not emitted by instrumentation (because SDK respects it) by default — they only appear when the user explicitly opts in via Views. So there's nothing to exclude in the zero-config case.

For exclude_attributes, the instrumentation library itself should stop emitting them (or the semantic conventions should guide the migration). The fix belongs at the instrumentation layer, not as an SDK-side exclude hint.

The existing Attributes (include) advisory already covers the zero-config story: the instrumentation author lists the recommended attribute keys, and anything not on that list is implicitly not recommended.

[dashpole]

As @cijothomas points out, this is already supported by the Attributes advisory parameter.

[cijothomas]

Separately, I realized that Declarative Config already includes support for this - https://github.com/open-telemetry/opentelemetry-configuration/blob/main/schema/meter_provider.yaml#L605-L609 Spec wording is a bit hand-wavy/flexible, so I think it makes sense to make it more concrete.

[dashpole]

That is part of the SDK's View, not part of the API.

[reyang]

@thompson-tomo is there an issue? We don't take new features unless there is an issue and the issue is triaged and accepted.

https://github.com/open-telemetry/opentelemetry-specification/blob/main/CONTRIBUTING.md#proposing-a-change

[reyang]

Need an issue + follow the issue workflow https://github.com/open-telemetry/opentelemetry-specification/blob/main/CONTRIBUTING.md#proposing-a-change.

[thompson-tomo]

@cijothomas / @dashpole I have descoped this PR to only touch the sdk view config.

This pr now bring across the definition from declarative config ie include & exclude of attributes. I have left the existing attribute.

[thompson-tomo]

@reyang i was asked to raise a pr for this by a spec sponser @cijothomas as per https://cloud-native.slack.com/archives/C01N7PP1THC/p1773713324834269?thread_ts=1773667594.006789&channel=C01N7PP1THC&message_ts=1773713324.834269 hence I didn't think an issue was needed.

As identified by the sponser this functionality is actually already defined in stable declarative config so is it actually a new feature? And at the same time the spec already mentioned attributes can be excluded.

[reyang]

As identified by the sponser this functionality is actually already defined in stable declarative config so is it actually a new feature? And at the same time the spec already mentioned attributes can be excluded.

This is what I was using to determine for this PR:

1. does this PR has "feat" in the title - if yes, I assume it is a feature
2. does the changelog mention anything like "add" - if yes, I assume it is a feature

[thompson-tomo]

Ok, I understand but in the case here it could just as easily be classified as a fix given that the spec itself mentions the ability to exclude attributes and it is in released stable declarative config. If you really want it could be classified as a corrective feature given it is fixing 2 things.

Anyway the issue is #4960

[github-actions]

This PR was marked stale. It will be closed in 14 days without additional activity.

[thompson-tomo]

Awaiting @jack-berg for confirmation about alignment with declarative config.

[jack-berg]

Declarative config has a policy where we only model properties described in the spec or semantic conventions.

My understanding is that this PR is trying to address an apparent discrepancy between the declarative config schema and the spec with respect to a view stream's attribute_keys.

Some context:

• Here's the PR where declarative config switched just included to included + excluded. Notably, we blocked it until a spec change was made to explicitly add exclude language (Add support for filtering attribute keys for streams via an exclude list #4188).
• Here's a PR where we went from informally defining the semantics of properties in comments on examples, to defining them as part of the schema, with build checks to make sure all properties were described. Before this PR, ViewStream.attribute_keys was the generic type IncludeExclude, but the semantics of IncludeExclude where different based on usage. For example, in resource detection, include / exclude had glob matching semantics, while in view streams include / exclude had no such mention.
• The current text of the spec makes explicit mention of an allow list of keys (i.e. declarative config's IncludeExclude.included) and explicit mention of an excluded list (i.e. declarative config's IncludeExclude.excluded. The property names used in declarative config don't exactly match the spec, but that's less important than the concepts matching. I will note there is a real discrepancy in that the declarative config IncludeExclude has glob pattern matching semantics, where there is no reference of this in the view spec. I write this miss up to an accidental oversight going as we we from informally capturing property semantics in example comments to formally capturing as part of the schema. Not ideal, but in this case, not harmful in practice.

@thompson-tomo from my POV, the only discrepancy between the spec and declarative config is the glob pattern matching semantics. I think declarative config modeling attribute_keys as a type called IncludeExclude with properties included and excluded is a reasonable data modeling decision given the language in the spec.

[thompson-tomo]

Thanks @jack-berg for me you have missed the discripency.

The spec for attribute_keys states:

This is, at a minimum, an allow-list of attribute keys for measurements captured in the metric stream. The allow-list contains attribute keys that identify the attributes that MUST be kept, and all other attributes MUST be ignored.

That differs to declarative config:

Configure attribute keys retained in the resulting stream(s).

If we look at the most common use case, I want to opt-in to an optional attribute. based on the current spec I would need to know all other attributes so that I can list them along with the single attribute to add with the attribute property.

This changes it so that we are able to just specify the attribute to retain in addition to the standard attributes via the include property.

What we could do instead is one of the below but I am not if they would be considered breaking.

• add an explicit exclude_attribute_keys property & change the description of the attribute_keys property so it doesn't ignore not defined attributes.
• extend previous option to include a flag property to drop unknown to preserve existing use case of only including attributes specified.
• extend first option to mention that to drop unspecified attributes, set exclude property to have a value of *.
• deprecate attribute_keys and introduce the include/exclude property which behaves as per previous option

[jack-berg]

If we look at the most common use case, I want to opt-in to an optional attribute. based on the current spec I would need to know all other attributes so that I can list them along with the single attribute to add with the attribute property.

Sorry this goal was not clear to me. You should put this prominently in the PR title / description.

Declarative config's stream config does not accomplish this. There's no mention of ViewStream.attribute_keys.included defining the set of attribute keys which must be retained in addition to the metric advisory parameter.

This is net new behavior.

From a declarative config standpoint, changing the semantics of ViewStream.attribute_keys.included to be additive to the metric advisory parameter attributes would be a breaking change. We'd need to introduce a new property to accommodate this, which is allowed under the versioning policy.

[thompson-tomo]

Declarative config's stream config does not accomplish this. There's no mention of ViewStream.attribute_keys.included defining the set of attribute keys which must be retained in addition to the metric advisory parameter.

You are right it doesn't mention that, however it doesn't mention that all other attributes will be excluded. In fact I take it as implicit inclusion due to semconv defining them as included.

What it does say

If omitted, all values are included.

However that statement is problematic when we look at Resource Detection which uses the same object/definition as that would make all opt-in attributes in semconv now the same as recommended with the opt-out nature rather than opt-in. We don't want that.

So we could:

• address the grey area in declarative config by making explicit. Perhaps something like:

If omitted, values are included based on the telemetry definition if found otherwise all attributes are included.

• add declarative config property dropNonIncluded so we can cater for that use case.
• still update the spec as it currently does not even define an exclude parameter just mentions it in free text.

As an aside it would be beneficial to persue automatic documenting of configuration of options for semconv as I attempted to do with open-telemetry/semantic-conventions#3396 but got no traction.

[jack-berg]

You are right it doesn't mention that, however it doesn't mention that all other attributes will be excluded. In fact I take it as implicit inclusion due to semconv defining them as included.

Let's discuss through some examples:

Recorded Attribute Keys	Attribute Keys Advisory Param	ViewStream.attribute_keys.included	ViewStream.attribute_keys.excluded	Resolved Attribute Keys
foo,bar,baz	unset	unset	unset	foo,bar,baz
foo,bar,baz	foo,bar	unset	unset	foo,bar
foo,bar,baz	foo,bar	unset	*	[]
foo,bar,baz	foo,bar	unset	foo	bar,baz
foo,bar,baz	foo,bar	foo	unset	foo
foo,bar,baz	foo,bar	foo,b*	bar	foo,baz
foo,bar,baz	foo,bar	foo

There's no interaction with the Attribute Keys Advisory Param, except when included / excluded are unset. If either included and/or excluded are set, they combine to form a predicate (Predicate in java lingo) which attribute keys are evaluated against.

However that statement is problematic when we look at Resource Detection

IncludeExclude does apply to resource to detection as well via .resource.detection/development.attributes, but the filter it forms applies to the output of .resource.detection/development.detectors, which is operates on an opt-in basis (i.e. wysiwyg - what you see is what you get).

Let's use another example to discuss:

resource:
  attributes:
    - name: my.resource.attribute
       value: value
  detection/development:
    detectors:
      - service: // adds service.instance.id and populates service.name from OTEL_SERVICE_NAME
      - host: // adds host.* attributes
      - process: // adds process.* attributes
      - container: // adds container.* attributes
    attributes:
      included:
        - *
      excluded:
        - process.command_line

• The names of well known resource detectors and their semantics are defined in the spec here
• This is an SDK extension plugin interface, so can be extended with other resource detectors via the declarative config PluginComponentProvider mechanism
• Each detector can accept config, but none of the ones defined in the spec do, so we need .resource.detection/development.attributes to be able to exert some control on the output.
• Here we say that all resource detection attributes are included, but explicitly exclude process.command_line.
• The set of well known resource detectors defined in the spec is still in development, and so the whole .resource.detection/development block is in development as well and subject to breaking changes.

I don't view the current state of resource detection as ideal. The set of well known resource detectors was added extremely late (well after languages defined these things with their own subjective interpretations) and the semantics (i.e. which exact attributes they provide) are underdefined. This is a stop gap that will hopefully defined with entities, which I expect to have much more rigorously defined entity detectors defined in semantic conventions.

[thompson-tomo]

Those examples all work well, so I can see how the design was design has come to be. But they have all focussed on defining the view aspect as opposed to instrument creation as mentioned in https://opentelemetry.io/docs/specs/otel/metrics/sdk/#instrument-advisory-parameters, hence my example above:

If we look at the most common use case, I want to opt-in to an optional attribute.

This is based on examples in sem-conv such as https://opentelemetry.io/docs/specs/semconv/system/system-metrics/#metric-systemcputime among others.

If I replicate your table for that example to opt-in to the 1 attribute I get.

Recorded Attribute Keys	Attribute Keys Advisory Param	ViewStream.attribute_keys.included	ViewStream.attribute_keys.excluded	Resolved Attribute Keys
n/a	unset	cpu.mode, cpu.logical_number	unset	cpu.mode, cpu.logical_number

Hence all attributes must be known. If the example was more complex let's say https://opentelemetry.io/docs/specs/semconv/http/http-metrics/#metric-httpserverrequestduration and I wanted to include server.address but not user_agent.synthetic.type my config based on the above approach would need to have 8 attributes listed within attribute_keys.included.

In relation to resource detector the way I would interrupt that is include all attributes except for process.command_line which would mean that we would end up with host.ip/mac included even though it is opt-in https://opentelemetry.io/docs/specs/semconv/registry/entities/host/. How could I even include host.ip without listing every single attribute.

As an aside I think open-telemetry/weaver#1230 would help with atleast definining resource detectors in terms of their names & the entities provided.