Skip to content
Open
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
184 changes: 177 additions & 7 deletions docs/rfcs/configuration-merging-strategy.md
Original file line number Diff line number Diff line change
Expand Up @@ -258,15 +258,185 @@ otelcol --config=main.yaml --config=extra_components.yaml?merge_mode=append --co

- In the above command, we have no specified any options for `override_components.yaml`. Hence, it will override all the conflicting lists from previous configuration, which is the default behaviour.

### Approach 3: CLI flags (`--merge-list`)

This approach adds separate top-level CLI flags to control the merging behaviour, instead of adding it to config file URIs.

#### Design

Two new flags are added to the collector:

| Flag | Description |
|---|---|
| `--merge-list=<path>` | Mark a config path (glob pattern) as a merge target. Repeatable. |
| `--merge-mode=<append\|prepend>` | Merge mode. Default: `append`. |

`<path>` follows the same `::` separator convention already used in confmap (e.g. `service::extensions`, `service::pipelines::*::processors`).

#### Examples

1. _Merge all lists under `service`_:

```bash
otelcol \
--config=main.yaml \
--config=extra.yaml \
--merge-list="service::*" \
--feature-gates=confmap.enableMergeAppendOption
```

2. _Merge extensions (append) and processors (prepend) independently_:

```bash
otelcol \
--config=main.yaml \
--config=extra_extensions.yaml \
--config=extra_processors.yaml \
--merge-list="service::extensions:append" \
--merge-list="service::pipelines::*::processors:prepend" \
--feature-gates=confmap.enableMergeAppendOption
```

3. _Merge a deeply nested transform processor statement list_:

```bash
otelcol \
--config=main.yaml \
--config=extra_transforms.yaml \
--merge-list="processors::transform::statements" \
--feature-gates=confmap.enableMergeAppendOption
```

---

### Approach 4: `merge:` block inside a config file

Instead of annotating individual yaml nodes (Approach 1) or touching the command line (Approaches 2 & 3), a config file can carry a dedicated top-level `merge:` section that declares which paths in *that file* should be merged into the running configuration. The `merge:` section will be removed before the config is applied.

This keeps URIs opaque, keeps the command line short, and keeps merge intent with the data it governs.

#### Schema

```yaml
merge:
mode: append # append | prepend (default: append)
fields:
- "service::extensions"
- "service::pipelines::*::receivers"
```

#### Examples

1. _Append extensions from a supplementary file_:

```yaml
# main.yaml
extensions:
health_check:

service:
extensions: [health_check]
pipelines:
logs:
receivers: [otlp]
exporters: [debug]
```

```yaml
# extra_extension.yaml
extensions:
file_storage:

service:
extensions: [file_storage]

merge:
mode: append
fields:
- "service::extensions"
```

```yaml
# final (effective) configuration
extensions:
health_check:
file_storage:

service:
extensions: [health_check, file_storage]
pipelines:
logs:
receivers: [otlp]
exporters: [debug]
```

```bash
# No special flags needed — merge intent lives in the file
otelcol --config=main.yaml --config=extra_extension.yaml \
--feature-gates=confmap.enableMergeAppendOption
```

2. _Merge all pipeline lists (extensions + per-pipeline receivers/processors/exporters)_:

```yaml
# sidecar_fragment.yaml
extensions:
memory_limiter:
receivers:
prometheus/sidecar:

service:
extensions: [memory_limiter]
pipelines:
metrics:
receivers: [prometheus/sidecar]

merge:
mode: append
fields:
- "service::extensions"
- "service::pipelines::*::receivers"
- "service::pipelines::*::processors"
- "service::pipelines::*::exporters"
```

3. _Mix append and prepend in a single fragment_:

```yaml
# fragment.yaml
processors:
transform/enrich:
memory_limiter:

service:
pipelines:
traces:
processors: [memory_limiter] # prepend: run first
logs:
processors: [transform/enrich] # append: run last

merge:
fields:
- path: "service::pipelines::traces::processors"
mode: prepend
- path: "service::pipelines::logs::processors"
mode: append
```

---

## Open questions

- What to do if an invalid option is provided for `merge_mode` or `merge_paths`?
- I can think of two possibilities:
1. Error out.
2. Log an error and merge the default way
- What to do if an invalid query param is provided in config URI?
- In this case, I strongly feel that we should error out.
- What to do if an invalid option is provided for `merge_mode`, `merge_paths`, `--merge-list`, or `merge:` fields?
- Two possibilities:
1. Error out (recommended).
2. Log an error and merge the default way.
- **Approach 2**:What to do if an invalid query param is provided in a config URI?
- Strongly prefer erroring out.
- **Approach 3**: Should `--merge-list` apply to *all* loaded config files?
- **Approach 4**: Should `merge:` be allowed in the very first config file (where it would be a no-op), or should that produce a warning?
- **Approach 4**: What is the reserved key name? Alternatives: `_merge`, `x-merge`, `confmap_merge`.

## Extensibility

This URI-based approach is highly extensible. In the future, it can enable advanced operations such as map overriding. Currently, it's impossible to do so.
The URI-based approach (Approach 2) and the `merge:` block approach (Approach 4) are both highly extensible. In the future they can enable advanced operations such as map overriding. Currently those operations are impossible.
Loading