Add option to programmatically customization declarative config components, add supplementary guidance (#4777)

Resolves #4583.

We've had discussions in the declarative config SIG recently about how
programmatic configuration interacts with declarative config. Questions
of what should take priority, and whether there should be options to
programmatically customize (#4583) to address concepts which are not
expressible in declarative config.

The goal here is to avoid the type of spec ambiguity which resulted SDKs
having such different behavior in environment variable <> programmatic
configuration intersection. Declarative configuration is green field and
for a short time longer is still experimental, and so we are not bound
by historical constraints.

Reading through the relevant specs, I do not think there is any
ambiguity about configuration interface priority. `create` returns
resolved top-level SDK components. If a SDK supports updating
components, then the components returned by `create` will be subject to
whatever semantics the SDK already has decided for how updates occur to
SDK components programmatically created to begin with. To make this
clear, I opted to add a new supplementary guidelines document to give
advice to implementers without cluttering or adding more normative
language.

On a related note, I thought it was pertinent to address #4583
simultaneously, but adding explicit language to allow (i.e. normative
MAY) implementations of `create` to allow programmatic customization of
SDK components as they are initialized.

cc/ @open-telemetry/configuration-approvers, @JamieDanielson, @maryliag,
@pellared

---------

Co-authored-by: Marylia Gutierrez <marylia.gutierrez@grafana.com>

Author: jack-berg

CHANGELOG.md

@@ -65,6 +65,9 @@ release.
 
 - Clarifies that guidance related to boolean environment variables is not applicable
   to other configuration interfaces. ([#4723](https://github.com/open-telemetry/opentelemetry-specification/pull/4723))
+- Declarative configuration: add optional programmatic customization to
+  `create`, and add related supplemental guidelines.
+  ([#4777](https://github.com/open-telemetry/opentelemetry-specification/pull/4777))
 
 ## v1.51.0 (2025-11-17)
 

specification/configuration/README.md

@@ -55,6 +55,8 @@ Declarative configuration consists of the following main components:
   operations to parse configuration files and interpret the configuration data
   model.
 
+See also [supplementary guidelines](./supplementary-guidelines.md).
+
 ### Other Mechanisms
 
 Additional configuration mechanisms SHOULD be provided in whatever

specification/configuration/sdk.md

@@ -302,6 +302,29 @@ This SHOULD return an error if it encounters an error in `configuration` (i.e.
 fail fast) in accordance with
 initialization [error handling principles](../error-handling.md#basic-error-handling-principles).
 
+SDK implementations MAY provide options to allow programmatic customization of
+the components initialized by `Create`. This allows configuration of concepts which
+are not yet or may never be representable in the configuration model. For example,
+java OTLP exporters allow configuration of the [ExecutorService](https://docs.oracle.com/javase/8/docs/api/java/util/concurrent/ExecutorService.html),
+a niche but important option for applications which need strict control of thread pools.
+This programmatic customization might take the form of passing an optional callback to
+`Create`, invoked with each SDK subcomponent (or a subset of SDK component types) as
+they are initialized. For example, consider the following snippet:
+
+```yaml
+file_format: 1.0
+tracer_provider:
+  processors:
+    - batch:
+        exporter:
+          otlp_http:
+```
+
+The callback would be invoked with the SDK representation of an OTLP HTTP exporter, a
+Batch SpanProcessor, and a Tracer Provider. This pattern provides the opportunity
+to programmatically configure lower-level without needing to walk to a particular
+component from the resolved top level SDK components.
+
 TODO: define behavior if some portion of configuration model is not supported
 
 #### Register ComponentProvider

specification/configuration/supplementary-guidelines.md

@@ -0,0 +1,65 @@
+# Supplementary Guidelines
+
+Note: this document is NOT a spec, it is provided to support the declarative config
+[API](./api.md) and [SDK](./sdk.md) specifications, it does NOT add any extra
+requirements to the existing specifications.
+
+<details>
+<summary>Table of Contents</summary>
+
+<!-- toc -->
+
+- [Configuration interface prioritization and `create`](#configuration-interface-prioritization-and-create)
+- [Programmatic customization and `create`](#programmatic-customization-and-create)
+
+<!-- tocstop -->
+
+</details>
+
+## Configuration interface prioritization and `create`
+
+With the [environment variable](./sdk-environment-variables.md) configuration
+interface, the spec failed to answer the question of whether programmatic or
+environment variable configuration took precedence. This led to differences in
+implementations that were ultimately stabilized and difficult to resolve after
+the fact.
+
+With declarative config, we don't have ambiguity around configuration interface
+precedence:
+
+* [`parse`](./sdk.md#parse) is responsible for parsing config file contents and
+  returning the corresponding in-memory data model. Along the way, it
+  performs [environment variable substitution](./data-model.md#environment-variable-substitution).
+* [`create`](./sdk.md#create) is responsible for interpreting an in-memory
+  config data model and creating SDK components.
+
+There is no precedence ambiguity with the environment variable configuration
+interface: The language of `parse` and `create` is explicit about
+responsibilities and makes no mention of merging environment variables outside
+of environment variable substitution.
+Furthermore, [OTEL_EXPERIMENTAL_CONFIG_FILE](./sdk-environment-variables.md#declarative-configuration)
+explicitly states that the environment variable configuration scheme is ignored.
+
+There is no precedence ambiguity with
+the [programmatic configuration interface](./README.md#programmatic): `create`
+consumes an in-memory config data model and creates SDK components. According to
+the trace, metric, and log specs, SDKs MAY support updating the config, but
+there is no conflict with declarative config which doesn't already exist.
+However, the SDK handles programmatic config updates to SDK components which
+originally programmatically configured applies here as well. If an SDK supports
+it, all programmatic config updates are applied after `create` initializes SDK
+components and therefore take precedence. The semantics of what programmatic
+config updates are allowed and how they merge with existing SDK components are
+out of scope for declarative config.
+
+## Programmatic customization and `create`
+
+While `create` does provide an optional mechanism for programmatic
+customization, its use should be considered a code smell, to be addressed by
+improving the declarative config data model.
+
+For example, the fact that configuration of dynamic authentication for OTLP
+exporters is not possible to express with declarative config should not
+encourage the OpenTelemetry community to have better programmatic customization.
+Instead, we should pursue adding authentication as an SDK extension plugin
+interface and modeling this new plugin in declarative config.