[OTEP] Federate Semantic Conventions

[jsuereth]

DO NOT MERGE

This is a draft proposal to federate the semantic-conventions repository to allow for faster innovation and evolution across the ecosystem.

• We retain clear ownership of semantic convention domains
• We retain clear policy and tooling support for instrumentation authors to provide comprehensive documentation, stability guarantees and integration test assurance of the signals the produce.
• We provide a clear "shared" location for x-cutting attributes and definitions.
• We provide a path for federated semantic conventions to promote to core, if desired.
• We expand for non-opentelemetry sources of conventions and standards to advertise their instrumentation and leverage the same tooling as open-telemetry.

[lmolkova]

so that there is no chance of having conflicts / vendoring in in incompatible manner / etc?

I like it!

[lmolkova]

this is awesome!

[lmolkova]

nit: maybe explain what policies are (link to some section/doc), I imagine it's not a commonly known concept

[lmolkova]

nit: it's the same on as the last one? (policies/check/naming)

[lmolkova]

nit:

I believe we now have the opt-in mechanism that's slightly different than picking a schema url, it was just introduced in declarative config: https://github.com/open-telemetry/opentelemetry-configuration/blob/4351ebd2805746d047a23588f8d3abe89f40f79f/snippets/ExperimentalInstrumentation_kitchen_sink.yaml#L52

E.g.

instrumentation:
  general:
    rpc:
      semconv:
        version: 1
        experimental: false
        dual_emit: true

[lmolkova]

nit: it's not part of weaver.yaml now, right? do we want to make it?

Or should we ship GH actions and/or have some shareable workflows that repos would reuse or copy-paste?

[lmolkova]

Do we want to allow stable (federated) registry to depend on unstable parts of otel conventions?

I think we can go either way.
If we don't allow it: they can take dependency on stable parts only and vendor in unstable parts if necessary.
If we allow it, they should be able to communicate breaking changes in unstable core via semver.

Either way, we need some forcing factors for them to

• update to newer underlying core - it will be hard regardless
• bring common concepts to core conventions

I'd rather not allow it at least initially and encourage to vendor in experimental stuff.

[lmolkova]

this would probably be the forcing factor to update federated registries I mentined in prev comment.
If library A depends on semconv core v1.40 and library B on semconv core v1.140 they probably can't coherently work in the same distro.

[lmolkova]

nit: would we separate HTTP from core conventions? Probably not until it needs v2

[lmolkova]

This is awesome!

I also like the year as a major version - we should create expectation of some breaking changes on a predictable cadence

[mikeldking]

This is date ver no? This is not semver.

[lmolkova]

we should maintain a list of federated conventions in semconv (or somewhere) for discoverability. And using it we can even have a weekly check that validates all federated registries together to find conflicts or tests them individually against latest core. If some collisions are found, we could automatically create issues and notify maintainers about conflicts.

It would also be a good forcing factor for conventions to stay in sync with core when they can.

[jsuereth]

Added

[lmolkova]

[update] I see you have it addressed later

assuming it's not required, what would be the motivation to even start bringing semconv to core?

I can imagine these:

• offload maintenance of a stable thing to core repo
• need to align between different federated repos (e.g. align db server with db client)
• need to align compatibility for platform release

But unless something starts to break, I don't see people being too interested in this work. And I think it's usually fine, but it's likely that we'll grow into 5 different versions of some conventions across 5 different repos.

We should probably require sigs like GenAI to be language-agnostic to avoid it.

[jsuereth]

Absolutely agree. I think that's the inevitability of federating and we'll need to balance cost / benefit here. I expect we'll have two types of fedaration:

• jvm / go runtime like things where it's really a specific technology that needs to be addressed. Unlikely we'll need to merge this back in ever.
• GenAi, db, http type things where they need to be cross language and cross cutting. These main want to come back into core, so their attributes can be re-used in further federated registries without as much dependnecy hell, but do not need to.

[mikeldking]

How does this propagate to natively instrumented libraries and what incentive do they have to move to newer conventions? There always is a transform layer running?

[mikeldking]

are conventions in a particular version all uniformly considered "stable"?

[mikeldking]

what are the rules of semver here. Like dropping an attribute is "breaking" - what if it was optional?

[mikeldking]

What's the acceptance criteria for "tested together", what if an instrumentation fails, is it removed from release?

[mikeldking]

What if end-users want to opt out of expensive / PII risking conventions. Do these still land in core? What is the final say for something deserving to be in core. I'm unfamiliar here.

[jsuereth]

We allow expensive / PII in core, with opt_in today and there's guidance around it. I think this is an orthogonal concern to this proposal, but an important issue that needs a solution across various components of OpenTelemetry.

For this proposal, we continue with the current behavior we have.

[mikeldking]

this is a good example, but also centralized with the maintainers of Java. So it seems like a simpler example? It's harder to do this when there is no central vendor.

[mikeldking]

where to these actions run - on OTEL repos only? How does it run on native libraries.

[jsuereth]

So we only "enforce" this structure for OTEL-owned repositories/conventions.

For native libraries, we provide open-telemetry/weaver as a tool they can use to participate in this federation, but we do not enforce any structure / policy unless they are planning to move their definitions back into the OpenTelemetry project.

[mikeldking]

what if the federated conventions need the unstable conventions for it to make sense - example being session / threading of agents.

[jsuereth]

Then the federated convention would be marked "unstable" as well. Basically you can't declare a "child" convention stable unless the "parent" is also stable.

This doesn't preclude usage, just stability declaration.

[mikeldking]

dual_emit prioritizes which verion?

[jsuereth]

In OpenTelemetry - dual_emit will fire both conventions, i.e. you'll get both attributes, multiple metrics, etc.

@trask or @lmolkova may be able to speak more to how that works in practice.

[Aneurysm9]

What does "both conventions" mean, though? Some prior version of the jvm federated registry? The vestigial conventions in the semconv registry?

[mikeldking]

which users are we talking about. The end-user?

[jsuereth]

Yes

[mikeldking]

this is not so simple. Instrumentation is dependent on now two versions, the library it instruments AND the semcov that it tracks. What if the instrumentation needs to move a major version for library changes, then subsequently needs to backfill support for an older version. Now there is no tenable way to track both semcov breaking changes with library breaking changes.

[jsuereth]

I think I intend this to mean the opposite, or at least I don't see them as coupled.

• If you make a breaking change in semconv, you need a major version bump on your library.
• You may need major version bump on your library for other reasons, you do NOT need to bump semconv major version for this.

The key here is folks understand they need to do the first, and that the output signals of an instrumentation library are part of its stability.

Regarding tracking both semconv breaking + library breaking changes - I wanted to keep this simpler where you mostly just pay attention to library major version bumps, and then schema_url and backends can help you handle incompatibilities / conversions downstream.

[mikeldking]

What if the library is unstable?

[jsuereth]

We would allow semconv to be marked as stable prior to libraries being fully stable. This is more to prevent the opposite in otel - libraries which are production-ready NOT declaring stability, because of a looming threat of semconv changes.

[mikeldking]

the problem here is "cost" - the cost to one OTLP backend might be significant - it dictates a business deal being made in a quarter vs losing to a competitor that doesn't comply to Otel at all. This puts vendors that are all in on OTEL at risk as their upstream signals are being dictated by parties that have no actual financial overhead or incentives to move the standards forward.

[Aneurysm9]

I'm also not sure this aligns with the Contrib Stability section of the versioning and stability doc:

Plugins, instrumentation, and other contrib packages SHOULD be kept up to date and compatible with the latest versions of the API, SDK, and Semantic Conventions. If a release of the API, SDK, or Semantic Conventions contains changes which are relevant to a contrib package, that package SHOULD be updated and released in a timely fashion.

If the relevant federated semconv decides to ship new major versions every other Tuesday because they're post-1.x and yet have attributes or telemetry that are in flux then it seems like the instrumentation libraries that depend on it would also be expected to ship new major versions frequently. Maybe that's something we'd want the TC to put a stop to, but it does seem like an expected result of a combined push for "stable semver" and "faster evolution".

[jsuereth]

So the idea behind this federation is that the lifecycle of a federated semconv and the instrumentation are aligned. You wouldn't bump one without the other, but you can bump both without bumping core semconv versions or requiring syncing beyond that scope.

The owners of the instrumentation and the owners of the "federated semconv" would be the same set of folks.

The wording in the Contrib Stability section would be updated appropriately here as well.

[mikeldking]

how can we say that backends will use these agent-aided transformations? That seems like something that certain vendors such as companies that have privileged foundation model access a competitive advantage. This seems unfair.

[jsuereth]

Good point, and not the real intention here.

The agent-aided meant we may want to start leveraging agents within open-telemetry to define transitions at definition time, and publicizing these transitions for everyoen to consume without agents. We cannot have agent-in-the-loop on hot paths, that's intractable. Leveraging an agent to help improve the coverage of how many version transitions we can safely provide is what this was intended to mean.

The true goal is that we have an open model and easy-to-use transition "definition" that folks can consume to translate between versions of schema, or even (if we're successful) between different schema-urls entirely.

That's still experimental / long-term exploration for the project. For now, we have targeted version-bump transitions as something we believe we can provide automatic transition model that should NOT be costly unfair for the ecosystem to adopt.

[mikeldking]

I'm not familiar with the weaver MCP server. How does this work?

[jsuereth]

Today it's pretty light. You run weaver mcp and it will load in a registry and allow the agent to interact with a version of semantics and lookup the model, definitions, notes, etc. It can also run live-check to enforce conformance on the version. We hope to expand this further to be more of an aide when defining conventions and allow more ad-hoc agent-initiated flows, e.g. "Help me create a semantic convention registry from this integration test"

[github-actions]

This PR was marked stale. It will be closed in 14 days without additional activity.

[dashpole]

It would also be nice to think through where the code generators and generated code would live. Would the existing language generators also track and generate federated repos, or would they need to publish their generation artifacts (e.g. templates) in a way that is consumable (and stable?) for federated semconv publishers to use to generate their own stuff? Presumably we don't want federated publishers to also have to write their own code generators.

[jsuereth]

Code generators - https://github.com/open-telemetry/opentelemetry-weaver-packages

Generated code - perhaps this needs more though. My expectation was it would live beside the instrumentation that exists today. I think that should be sufficient.

[dashpole]

Got it. So stuff like https://github.com/open-telemetry/opentelemetry-go/tree/main/semconv/templates/registry/go would need to be replicated in each federated semconv area? Or is go the only ones that have our templates stored in our repo?

[github-actions]

This PR was marked stale. It will be closed in 14 days without additional activity.

[Aneurysm9]

What does "both conventions" mean, though? Some prior version of the jvm federated registry? The vestigial conventions in the semconv registry?

[Aneurysm9]

This feels like the opposite of how I understood the proposal in #4813. That seemed to say that instrumentation libraries should declare themselves stable without regard to the stability of the semantic conventions they use. This seems to be saying that semantic conventions used by stable instrumentation libraries must be, by definition, stable. What of an instrumentation library that is "stable" but for dependence on semantic conventions that are "experimental"?

[jsuereth]

You can see my back and forth about #4813 - That was the wrong impression and the wording (should be corrected).

The goal is that insrumetnation can declare stability prior to core semconv declaring stability, but that instrumentation must then maintain its own stability of what it produces. This is the mechanism for how.

[Aneurysm9]

Does making major version bumps when required count as remaining backward compatible? If not I'd fear that such a time-in-grade requirement could lead to intentional stagnation.

[jsuereth]

I'm not sure I parse what you are asking.

[Aneurysm9]

If the jvm conventions have been stable for 6mo and want to be merged into core, but recognize a need for a breaking change can they do so with a MV bump without resetting the clock on that 1y timer? Put another way, does "Reach stable status, and remain backwards compatible for a full year" mean the highest MV must be >1 and at least one year old or does it mean that v1.0 must be at least one year old and any breaking changes appropriately incremented the MV? If the former, would they be incentivized to sit on a change for 6mo to get v1.x into core before releasing v2.x? Is that the incentive we want?

[Aneurysm9]

I'm also not sure this aligns with the Contrib Stability section of the versioning and stability doc:

Plugins, instrumentation, and other contrib packages SHOULD be kept up to date and compatible with the latest versions of the API, SDK, and Semantic Conventions. If a release of the API, SDK, or Semantic Conventions contains changes which are relevant to a contrib package, that package SHOULD be updated and released in a timely fashion.

If the relevant federated semconv decides to ship new major versions every other Tuesday because they're post-1.x and yet have attributes or telemetry that are in flux then it seems like the instrumentation libraries that depend on it would also be expected to ship new major versions frequently. Maybe that's something we'd want the TC to put a stop to, but it does seem like an expected result of a combined push for "stable semver" and "faster evolution".