diff --git a/content/en/docs/zero-code/obi/_index.md b/content/en/docs/zero-code/obi/_index.md index 843e5e029499..3a31d20c502d 100644 --- a/content/en/docs/zero-code/obi/_index.md +++ b/content/en/docs/zero-code/obi/_index.md @@ -9,7 +9,6 @@ cascade: OTEL_RESOURCE_ATTRIBUTES_APPLICATION: obi OTEL_RESOURCE_ATTRIBUTES_NAMESPACE: obi OTEL_RESOURCE_ATTRIBUTES_POD: obi -cSpell:ignore: asyncio uvloop --- OpenTelemetry libraries provide telemetry collection for popular programming @@ -40,11 +39,13 @@ OBI offers the following features: - **Visibility into encrypted communications**: Capture transactions over TLS/SSL without decryption - **Context propagation**: Propagate trace context across services automatically -- **Protocol support**: HTTP/S, gRPC, gRPC-Web, MQTT, Memcached, and more +- **Protocol support**: HTTP/S, gRPC, gRPC-Web, JSON-RPC, MQTT, Memcached, and + more - **Database instrumentation**: PostgreSQL (including pgx driver), MySQL, MongoDB, Redis, Couchbase (N1QL/SQL++ and KV protocol) -- **GenAI instrumentation**: Trace and metrics for OpenAI and Anthropic Claude - API calls with automatic payload extraction +- **GenAI instrumentation**: Trace and metrics for OpenAI, Anthropic Claude, + Google AI Studio (Gemini), and AWS Bedrock API calls with automatic payload + extraction - **Low cardinality metrics**: Prometheus-compatible metrics with low cardinality for cost reduction - **Network observability**: Capture network flows between services with @@ -54,36 +55,32 @@ OBI offers the following features: - **Collector integration**: Run OBI as an OpenTelemetry Collector receiver component -## Recent highlights (v0.7.0) - -OBI v0.7.0 introduces several significant improvements: - -- **StatsO11y**: New statistical metrics pipeline for host-level network - statistics, starting with TCP RTT metrics -- **Expanded protocol coverage**: Added Memcached protocol tracing support -- **Enhanced GenAI instrumentation**: Added support for Anthropic Claude with - automatic payload extraction -- **Python asyncio improvements**: Added context propagation support for Python - asyncio workloads using `uvloop` -- **New example scenario**: Added an NGINX example covering direct routing, - reverse proxying, and route-based telemetry across Docker, Kubernetes, and - standalone deployments -- **Prometheus exemplars**: Support for exemplars in metrics export for better - correlation with traces -- **New span types**: SQL instrumentation now emits server spans for database - calls -- **Operational controls**: Added configurable `log_format` and Kubernetes API - reconnect interval settings -- **Network diagnostics**: Added `obi_bpf_network_ignored_packets_total` for - troubleshooting dropped network packets -- **Release artifacts**: CycloneDX SBOMs now included in release packages for - supply chain security +## Recent highlights (v0.8.0) + +OBI v0.8.0 expands protocol coverage, payload extraction, and deployment +documentation: + +- **Generic Go tracing improvements**: Added generic Go protocol support, + including context propagation for generic requests +- **Expanded protocol coverage**: Added JSON-RPC support across all languages +- **Deeper HTTP payload extraction**: Added full HTTP body extraction, with + bounded decompression for response bodies +- **Broader GenAI coverage**: Added Google AI Studio (Gemini) and AWS Bedrock + payload extraction, and fixed Vertex AI Gemini support +- **Named CIDR labels**: Network metrics can now label configured CIDR ranges + with human-readable names +- **New example scenario**: Added an Apache HTTP Server example alongside the + existing NGINX walkthroughs +- **Support documentation**: Added a project support matrix for release + artifacts and supported environments For a complete list of changes and upgrade notes, see the -[release notes](https://github.com/open-telemetry/opentelemetry-ebpf-instrumentation/releases/tag/v0.7.0). +[release notes](https://github.com/open-telemetry/opentelemetry-ebpf-instrumentation/releases/tag/v0.8.0). -If you want to explore the new NGINX example, see the -[example walkthrough](https://github.com/open-telemetry/opentelemetry-ebpf-instrumentation/tree/v0.7.0/examples/nginx). +If you want to explore the upstream examples, see the +[NGINX walkthrough](https://github.com/open-telemetry/opentelemetry-ebpf-instrumentation/tree/v0.8.0/examples/nginx) +and the +[Apache walkthrough](https://github.com/open-telemetry/opentelemetry-ebpf-instrumentation/tree/v0.8.0/examples/apache). ## How OBI works diff --git a/content/en/docs/zero-code/obi/configure/export-data.md b/content/en/docs/zero-code/obi/configure/export-data.md index bb2f9618001d..631634ee543e 100644 --- a/content/en/docs/zero-code/obi/configure/export-data.md +++ b/content/en/docs/zero-code/obi/configure/export-data.md @@ -244,7 +244,7 @@ The list of instrumentation areas OBI can collect data from: - `mqtt`: MQTT publish/subscribe message metrics (MQTT 3.1.1 and 5.0) - `couchbase`: Couchbase N1QL/SQL++ query metrics and KV (Key-Value) protocol metrics based on memcached protocol -- `genai`: GenAI client metrics (OpenAI and Anthropic) +- `genai`: GenAI client metrics (OpenAI, Anthropic, Gemini, and AWS Bedrock) - `gpu`: GPU performance metrics - `mongo`: MongoDB client call metrics - `dns`: DNS query metrics @@ -304,7 +304,7 @@ The list of instrumentation areas OBI can collect data from: - `mqtt`: MQTT publish/subscribe message traces (MQTT 3.1.1 and 5.0) - `couchbase`: Couchbase N1QL/SQL++ query traces and KV (Key-Value) protocol traces, with query text and operation details -- `genai`: GenAI client traces (OpenAI and Anthropic) +- `genai`: GenAI client traces (OpenAI, Anthropic, Gemini, and AWS Bedrock) - `gpu`: GPU performance traces - `mongo`: MongoDB client call traces - `dns`: DNS query traces @@ -558,7 +558,7 @@ The list of instrumentation areas OBI can collection data from: - `kafka`: Kafka client/server message queue metrics - `mqtt`: MQTT publish/subscribe message metrics - `couchbase`: Couchbase N1QL/SQL++ query metrics and KV protocol metrics -- `genai`: GenAI client metrics (OpenAI and Anthropic) +- `genai`: GenAI client metrics (OpenAI, Anthropic, Gemini, and AWS Bedrock) For example, setting the `instrumentations` option to: `http,grpc` enables the collection of `HTTP/HTTPS/HTTP2` and `gRPC` application metrics, and disables diff --git a/content/en/docs/zero-code/obi/configure/metrics-traces-attributes.md b/content/en/docs/zero-code/obi/configure/metrics-traces-attributes.md index 157ad5d1ee1e..1cfeff330e41 100644 --- a/content/en/docs/zero-code/obi/configure/metrics-traces-attributes.md +++ b/content/en/docs/zero-code/obi/configure/metrics-traces-attributes.md @@ -82,10 +82,10 @@ YAML section: `ebpf` You can configure the component under the `ebpf` section of your YAML configuration or via environment variables. -| YAML
environment variable | Description | Type | Default | -| ---------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | -------- | -| `context_propagation`
`OTEL_EBPF_BPF_CONTEXT_PROPAGATION` | Controls trace context propagation method. Accepted: `all`, `headers`, `ip`, `disabled`. For more information, refer to the [context propagation section](#context-propagation). | string | disabled | -| `track_request_headers`
`OTEL_EBPF_BPF_TRACK_REQUEST_HEADERS` | Track incoming `Traceparent` headers for trace spans. For more information, refer to the [track request headers section](#track-request-headers). | boolean | false | +| YAML
environment variable | Description | Type | Default | +| ---------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------- | -------- | +| `context_propagation`
`OTEL_EBPF_BPF_CONTEXT_PROPAGATION` | Controls trace context propagation method. Accepted: `all`, `headers`, `tcp`, `headers,tcp`, `disabled`. For more information, refer to the [context propagation section](#context-propagation). | string | disabled | +| `track_request_headers`
`OTEL_EBPF_BPF_TRACK_REQUEST_HEADERS` | Track incoming `Traceparent` headers for trace spans. For more information, refer to the [track request headers section](#track-request-headers). | boolean | false | ### Context propagation @@ -102,17 +102,21 @@ that also use TC must chain correctly with OBI. For more information about chaining programs, see the [Cilium compatibility documentation](../../cilium-compatibility/). -You can disable the TCP/IP level encoding and TC programs by setting -`context_propagation="headers"`. This context propagation is fully compatible -with any OpenTelemetry distributed tracing library. +You can disable the TCP-level propagation and Linux Traffic Control programs by +setting `context_propagation="headers"`. This mode is fully compatible with any +OpenTelemetry distributed tracing library. Context propagation values: -- `all`: Enable both HTTP and IP options context propagation +- `all`: Enable both HTTP header and TCP context propagation - `headers`: Enable context propagation via the HTTP headers only -- `ip`: Enable context propagation via the IP options field only +- `tcp`: Enable context propagation via the TCP packet path only +- `headers,tcp`: Enable both methods explicitly - `disabled`: Disable trace context propagation +`http` is accepted as an alias for `headers`, but `headers` is the preferred +name in examples and configuration. + To use this option in containerized environments (Kubernetes and Docker), you must: @@ -121,7 +125,7 @@ must: path - Grant the `CAP_NET_ADMIN` capability to the OBI container -gRPC and HTTP/2 are not supported. +gRPC and HTTP/2 are not supported for this network-level mode. For an example of how to configure distributed traces in Kubernetes, see our [Distributed traces with OBI](../../distributed-traces/) guide. @@ -154,22 +158,24 @@ because it can create false positives, for example, if an application sends SQL text for logging through a TCP connection. Currently, OBI natively supports the PostgreSQL and MySQL binary protocols. -### HTTP header enrichment for spans +### HTTP header and body enrichment for spans {#http-header-enrichment-for-spans} + +### HTTP header and body enrichment for spans -OBI can attach selected HTTP headers to spans through the -`ebpf.payload_extraction.http.enrichment` configuration section. This is useful -when you want to carry business or routing headers into traces without manually -instrumenting the application. +OBI can attach selected HTTP headers and selected HTTP body fields to spans +through the `ebpf.payload_extraction.http.enrichment` configuration section. +This is useful when you want to carry business or routing headers into traces +without manually instrumenting the application. The enrichment engine is rule-based: -- Set `enabled: true` to activate HTTP header enrichment. -- Use `policy.default_action` to define whether unmatched headers are included - or excluded. The default is `exclude`. -- Use `policy.match_order` to control rule evaluation. The default is - `first_match_wins`. -- Use `obfuscate` rules to redact sensitive values while still exposing the - header key on the span. +- Set `enabled: true` to activate HTTP header and body enrichment. +- Use `policy.default_action.headers` and `policy.default_action.body` in YAML + to define whether unmatched headers or body content are included or excluded. + The default for both is `exclude`. +- Use `obfuscate` rules to redact sensitive header values or JSON body fields + before they are attached to spans. +- Rules are evaluated in order. For example: @@ -182,8 +188,9 @@ ebpf: enrichment: enabled: true policy: - default_action: exclude - match_order: first_match_wins + default_action: + headers: exclude + body: exclude obfuscation_string: '***' rules: - action: obfuscate @@ -202,17 +209,40 @@ ebpf: - X-Custom-* - X-Dice-Roll case_sensitive: false + - action: include + type: body + scope: request + match: + methods: [POST] + url_path_patterns: + - /v1/chat/completions + - action: obfuscate + type: body + scope: request + match: + methods: [POST] + url_path_patterns: + - /v1/chat/completions + obfuscation_json_paths: + - $.messages[*].content ``` -The following environment variables control the policy defaults: +The following environment variables control the global enrichment behavior: - `OTEL_EBPF_HTTP_ENRICHMENT_ENABLED` -- `OTEL_EBPF_HTTP_ENRICHMENT_DEFAULT_ACTION` -- `OTEL_EBPF_HTTP_ENRICHMENT_MATCH_ORDER` - `OTEL_EBPF_HTTP_ENRICHMENT_OBFUSCATION_STRING` -Rules themselves are configured in YAML. If you expect large headers, increase -`ebpf.buffer_sizes.http` so OBI can capture the relevant values. +The `policy.default_action.headers` and `policy.default_action.body` settings +are configured in YAML only; there are no environment variables for these +defaults. + +Rules themselves are configured in YAML. Header rules use `match.patterns` and +optional `case_sensitive`. Body rules use `match.url_path_patterns`, +`match.methods`, and `match.obfuscation_json_paths`. + +Body extraction requires HTTP payload capture. Increase `ebpf.buffer_sizes.http` +so OBI can capture the request or response bytes you want to enrich. The limit +applies independently to requests and responses. ## Instance ID decoration diff --git a/content/en/docs/zero-code/obi/distributed-traces.md b/content/en/docs/zero-code/obi/distributed-traces.md index aa3fa8199591..1c0a2d9454d4 100644 --- a/content/en/docs/zero-code/obi/distributed-traces.md +++ b/content/en/docs/zero-code/obi/distributed-traces.md @@ -93,6 +93,9 @@ from OpenTelemetry SDK instrumented services still works. gRPC and HTTP/2 are not supported at network level. +If you need finer control, `context_propagation` also accepts `headers`, `tcp`, +and `headers,tcp`. `http` is accepted as an alias for `headers`. + This type of context propagation works for any programming language and doesn't require that OBI runs in `privileged` mode or has `CAP_SYS_ADMIN` granted. For more details, see the diff --git a/content/en/docs/zero-code/obi/network/config.md b/content/en/docs/zero-code/obi/network/config.md index e4f3c6cef798..05a8f13abe84 100644 --- a/content/en/docs/zero-code/obi/network/config.md +++ b/content/en/docs/zero-code/obi/network/config.md @@ -80,9 +80,9 @@ socket filter to capture the network events. This mode doesn't conflict with Cilium CNI or other eBPF programs, which use the Linux Traffic Control egress and ingress filters. -| YAML | Environment variable | Type | Default | -| ------- | ------------------------- | -------- | ------- | -| `cidrs` | `OTEL_EBPF_NETWORK_CIDRS` | []string | (empty) | +| YAML | Environment variable | Type | Default | +| ------- | ------------------------- | -------------------- | ------- | +| `cidrs` | `OTEL_EBPF_NETWORK_CIDRS` | list of CIDR strings | (empty) | CIDRs list, to be set as the `src.cidr` and `dst.cidr` attribute with the entry that matches the `src.address` and `dst.address` respectively. @@ -93,8 +93,17 @@ address matches multiple CIDR definitions, the flow is decorated with the narrowest CIDR. As a result, you can safely add a `0.0.0.0/0` entry to group all the traffic that does not match any of the other CIDRs. -If you set this property via environment variable each entry must be separated -by a comma, for example: +In YAML, each entry can be either a plain CIDR string or an object with `cidr` +and `name` fields. The `OTEL_EBPF_NETWORK_CIDRS` environment variable accepts a +comma-separated list of CIDR strings only. For example: + +```yaml +network: + cidrs: + - cidr: 10.0.0.0/8 + name: cluster-internal + - 192.168.0.0/16 +``` ```sh OTEL_EBPF_NETWORK_CIDRS=10.0.0.0/8,192.168.0.0/16 diff --git a/content/en/docs/zero-code/obi/setup/docker.md b/content/en/docs/zero-code/obi/setup/docker.md index dc714b1fd37e..e92b21ff248e 100644 --- a/content/en/docs/zero-code/obi/setup/docker.md +++ b/content/en/docs/zero-code/obi/setup/docker.md @@ -44,7 +44,7 @@ You can verify the signature of the container image using the following commands: ```sh -export VERSION=v0.7.0 +export VERSION=v0.8.0 # Verify a release image from Docker Hub cosign verify --certificate-identity-regexp 'https://github.com/open-telemetry/opentelemetry-ebpf-instrumentation/' --certificate-oidc-issuer 'https://token.actions.githubusercontent.com' otel/ebpf-instrument:${VERSION} @@ -80,7 +80,7 @@ don't have one, you can use this [simple blog engine service written in Go](https://macias.info): ```sh -export VERSION=v0.7.0 +export VERSION=v0.8.0 docker run -p 18443:8443 --name goblog mariomac/goblog:dev ``` diff --git a/content/en/docs/zero-code/obi/trace-log-correlation.md b/content/en/docs/zero-code/obi/trace-log-correlation.md index 3d14908f0a12..851d684c0648 100644 --- a/content/en/docs/zero-code/obi/trace-log-correlation.md +++ b/content/en/docs/zero-code/obi/trace-log-correlation.md @@ -5,7 +5,7 @@ weight: 35 description: Learn how OBI correlates application logs with distributed traces for faster debugging and troubleshooting. -cSpell:ignore: BPFFS ringbuffer +cSpell:ignore: BPFFS PYTHONUNBUFFERED ringbuffer --- OpenTelemetry eBPF Instrumentation (OBI) correlates application logs with @@ -119,6 +119,18 @@ and `span_id` fields into JSON log objects: Plain text logs are passed through unchanged and are **not enriched** with trace context. +#### Runtime buffering limitations + +The log enricher only sees trace context when the log write happens on the +request-handling thread. Runtimes that buffer stdout asynchronously can break +this assumption. + +- Python in Docker commonly needs `PYTHONUNBUFFERED=1` +- .NET `Console.Out` is buffered by default when stdout is a pipe; use a + `StreamWriter` with `AutoFlush = true` +- ASP.NET Core's default `Microsoft.Extensions.Logging.AddConsole()` pipeline is + not compatible because it writes from a background thread + ### 2. Trace export and log enrichment enabled Traces must be exported and log enrichment enabled: diff --git a/static/refcache.json b/static/refcache.json index 73185a22a38e..422a65213f8c 100644 --- a/static/refcache.json +++ b/static/refcache.json @@ -11419,6 +11419,10 @@ "StatusCode": 206, "LastSeen": "2026-04-03T18:40:19.553351857Z" }, + "https://github.com/open-telemetry/opentelemetry-ebpf-instrumentation/releases/tag/v0.8.0": { + "StatusCode": 206, + "LastSeen": "2026-04-17T18:11:47.603377278Z" + }, "https://github.com/open-telemetry/opentelemetry-ebpf-instrumentation/tree/2feaeb84ef7aeeb1e57e642b46c96b965ae4b804/internal/test/integration/k8s/manifests?from_branch=main": { "StatusCode": 206, "LastSeen": "2026-03-17T09:54:42.999934807Z" @@ -11435,6 +11439,14 @@ "StatusCode": 206, "LastSeen": "2026-04-03T19:48:49.65041184Z" }, + "https://github.com/open-telemetry/opentelemetry-ebpf-instrumentation/tree/v0.8.0/examples/apache": { + "StatusCode": 206, + "LastSeen": "2026-04-17T18:11:56.141549709Z" + }, + "https://github.com/open-telemetry/opentelemetry-ebpf-instrumentation/tree/v0.8.0/examples/nginx": { + "StatusCode": 206, + "LastSeen": "2026-04-17T18:11:52.877605927Z" + }, "https://github.com/open-telemetry/opentelemetry-ebpf-profiler": { "StatusCode": 206, "LastSeen": "2026-04-10T09:57:36.748982346Z"