docs: add Kubernetes Arize AX tracing guide by danehans · Pull Request #988 · agentgateway/website

danehans · 2026-08-28T17:32:07Z

Company

Not applicable; this pull request updates documentation.

Contribution summary

Add Kubernetes Arize AX tracing guides under LLM observability.
Publish matching guides in the stable and development documentation trees.
Document Secret-backed OTLP/HTTP and OTLP/gRPC setup, verification, and troubleshooting.

Checklist

Logo file

Not applicable to this documentation change.

Listing

Added Arize AX to the Kubernetes LLM observability integrations.

Preview

Both transport configurations were verified end to end on Kind with
agentgateway 1.5.0, and the full Hugo build completed successfully.

Signed-off-by: Daneyon Hansen <daneyon.hansen@solo.io>

cloudflare-workers-and-pages · 2026-08-28T17:37:56Z

Deploying agentproxy with Cloudflare Pages

Latest commit:	`ac75720`
Status:	✅ Deploy successful!
Preview URL:	https://354cf635.agentproxy.pages.dev
Branch Preview URL:	https://arize-ax-kubernetes-tracing.agentproxy.pages.dev

View logs

Signed-off-by: Daneyon Hansen <daneyon.hansen@solo.io>

krisztianfekete · 2026-08-31T13:09:09Z

+        - name: span.name
+          expression: '"openai.chat"'


OTel semconv recommends our default format, which is {method} {path_match}. This would also overwrite other providers, so I'd remove it.

Removed the hardcoded span name from both examples so the semantic convention default is kept across providers.

krisztianfekete · 2026-08-31T13:09:39Z

+        - name: openinference.span.kind
+          expression: '"LLM"'


There can be other kinds as well.

Removed the hardcoded OpenInference span kind. Arize AX now maps the operation emitted in the native gen_ai attributes.

krisztianfekete · 2026-08-31T13:10:39Z

+        - name: llm.system
+          expression: llm.provider
+        - name: llm.model_name
+          expression: coalesce(llm.responseModel, llm.requestModel)
+        - name: llm.input_messages
+          expression: 'flattenRecursive(llm.prompt.map(c, {"message": c}))'
+        - name: llm.output_messages
+          expression: 'flattenRecursive(llm.completion.map(c, {"role": "assistant", "content": c}))'
+        - name: llm.token_count.prompt
+          expression: llm.inputTokens
+        - name: llm.token_count.completion
+          expression: llm.outputTokens
+        - name: llm.token_count.total
+          expression: llm.totalTokens


IIRC we already emit these by default.

Confirmed. Removed the duplicate provider, model, and token aliases and documented that Arize AX natively maps the gen_ai attributes emitted by agentgateway.

krisztianfekete · 2026-08-31T13:12:00Z

+        - name: llm.input_messages
+          expression: 'flattenRecursive(llm.prompt.map(c, {"message": c}))'
+        - name: llm.output_messages
+          expression: 'flattenRecursive(llm.completion.map(c, {"role": "assistant", "content": c}))'


Can we call it out that this will ship PII into a vendor SaaS? I'd remove them from the default example and just document how to set it to avoid accidents.

Removed prompt and response content from the default examples. Added an explicit opt-in section with a warning about PII, credentials, and other sensitive data being sent to a third-party SaaS platform.

Signed-off-by: Daneyon Hansen <daneyon.hansen@solo.io>

…racing Signed-off-by: Daneyon Hansen <daneyon.hansen@solo.io>

Signed-off-by: Daneyon Hansen <daneyon.hansen@solo.io>

Nadine2016 · 2026-09-03T17:10:28Z

+   kind: Secret
+   metadata:
+     name: arize-credentials
+     namespace: agentgateway-system


Suggested change

namespace: agentgateway-system

namespace: {{< reuse "agw-docs/snippets/namespace.md" >}}

Nadine2016 · 2026-09-03T17:11:33Z

+Choose either OTLP/HTTP or OTLP/gRPC. Each option creates the following resources.
+
+- An `AgentgatewayBackend` that connects to Arize AX over TLS and reads the authentication headers from the `arize-credentials` Secret.
+- An `AgentgatewayPolicy` that exports sampled LLM traces from the `agentgateway-proxy` Gateway.


Suggested change

- An `AgentgatewayPolicy` that exports sampled LLM traces from the `agentgateway-proxy` Gateway.

- An `{{< reuse "agw-docs/snippets/policy.md" >}}` that exports sampled LLM traces from the `agentgateway-proxy` Gateway.

Nadine2016 · 2026-09-03T17:12:24Z

+          key: space-id
+---
+apiVersion: agentgateway.dev/v1alpha1
+kind: AgentgatewayPolicy


Suggested change

kind: AgentgatewayPolicy

kind: {{< reuse "agw-docs/snippets/policy.md" >}}

Nadine2016 · 2026-09-03T17:13:20Z

+          name: arize-credentials
+          key: space-id
+---
+apiVersion: agentgateway.dev/v1alpha1


Suggested change

apiVersion: agentgateway.dev/v1alpha1

apiVersion: {{< reuse "agw-docs/snippets/api-version.md" >}}

Nadine2016 · 2026-09-03T17:13:58Z

+kind: AgentgatewayBackend
+metadata:
+  name: arize-otlp
+  namespace: agentgateway-system


Suggested change

namespace: agentgateway-system

namespace: {{< reuse "agw-docs/snippets/namespace.md" >}}

Nadine2016 · 2026-09-03T17:14:11Z

+| EU | `otlp.eu-west-1a.arize.com` |
+| Canada | `otlp.ca-central-1a.arize.com` |
+
+The following examples use the US collector. Replace `spec.static.host` with the collector host for your region.


save this in an env also

Nadine2016 · 2026-09-03T17:14:32Z

+  namespace: agentgateway-system
+spec:
+  static:
+    host: otlp.arize.com


Suggested change

host: otlp.arize.com

host: ${ARIZE_HOST}

Nadine2016 · 2026-09-03T17:14:51Z

+kind: AgentgatewayPolicy
+metadata:
+  name: arize-tracing
+  namespace: agentgateway-system


Suggested change

namespace: agentgateway-system

namespace: {{< reuse "agw-docs/snippets/namespace.md" >}}

Nadine2016 · 2026-09-03T17:15:06Z

+kind: AgentgatewayBackend
+metadata:
+  name: arize-otlp
+  namespace: agentgateway-system


Suggested change

namespace: agentgateway-system

namespace: {{< reuse "agw-docs/snippets/namespace.md" >}}

Nadine2016 · 2026-09-03T17:15:13Z

+kind: AgentgatewayPolicy
+metadata:
+  name: arize-tracing
+  namespace: agentgateway-system


Suggested change

namespace: agentgateway-system

namespace: {{< reuse "agw-docs/snippets/namespace.md" >}}

Nadine2016 · 2026-09-03T17:15:28Z

+          key: space-id
+---
+apiVersion: agentgateway.dev/v1alpha1
+kind: AgentgatewayPolicy


Suggested change

kind: AgentgatewayPolicy

kind: {{< reuse "agw-docs/snippets/policy.md" >}}

Nadine2016 · 2026-09-03T17:15:46Z

+          name: arize-credentials
+          key: space-id
+---
+apiVersion: agentgateway.dev/v1alpha1


Suggested change

apiVersion: agentgateway.dev/v1alpha1

apiVersion: {{< reuse "agw-docs/snippets/api-version.md" >}}

Nadine2016 · 2026-09-03T17:26:05Z

+
+## Optional: Export LLM inputs and outputs
+
+The default configuration does not export prompt or response content. To display structured input and output messages in Arize AX, add the following attributes to the `AgentgatewayPolicy` for your selected transport.


Suggested change

The default configuration does not export prompt or response content. To display structured input and output messages in Arize AX, add the following attributes to the `AgentgatewayPolicy` for your selected transport.

The default configuration does not export prompt or response content. To display structured input and output messages in Arize AX, add the following attributes to the `{{< reuse "agw-docs/snippets/policy.md" >}}` for your selected transport.

Nadine2016 · 2026-09-03T17:26:23Z

+
+The default configuration does not export prompt or response content. To display structured input and output messages in Arize AX, add the following attributes to the `AgentgatewayPolicy` for your selected transport.
+
+{{< callout type="warning" >}}


Suggested change

{{< callout type="warning" >}}

Nadine2016 · 2026-09-03T17:26:42Z

+The default configuration does not export prompt or response content. To display structured input and output messages in Arize AX, add the following attributes to the `AgentgatewayPolicy` for your selected transport.
+
+{{< callout type="warning" >}}
+LLM prompts and responses can contain personally identifiable information (PII), credentials, or other sensitive data. Enabling these attributes sends that content to a third-party SaaS platform. Review your organization's data-handling requirements and configure appropriate guardrails or redaction before enabling them.


Suggested change

LLM prompts and responses can contain personally identifiable information (PII), credentials, or other sensitive data. Enabling these attributes sends that content to a third-party SaaS platform. Review your organization's data-handling requirements and configure appropriate guardrails or redaction before enabling them.

> [!IMPORTANT]

> LLM prompts and responses can contain personally identifiable information (PII), credentials, or other sensitive data. Enabling these attributes sends that content to a third-party SaaS platform. Review your organization's data-handling requirements and configure appropriate guardrails or redaction before enabling them.

Nadine2016 · 2026-09-03T17:26:56Z

+
+{{< callout type="warning" >}}
+LLM prompts and responses can contain personally identifiable information (PII), credentials, or other sensitive data. Enabling these attributes sends that content to a third-party SaaS platform. Review your organization's data-handling requirements and configure appropriate guardrails or redaction before enabling them.
+{{< /callout >}}


Suggested change

{{< /callout >}}

Nadine2016 · 2026-09-03T17:28:40Z

+
+Agentgateway supports custom OpenTelemetry resource attributes through `spec.frontend.tracing.resources`. Resource attributes are added to every exported span and can help you filter and group traces in Arize AX.
+
+In Kubernetes mode, agentgateway automatically sets `service.name`, `service.version`, `service.instance.id`, and `service.namespace`. The main configuration explicitly sets `deployment.environment.name`. You can add application-specific attributes such as `model_id` or `model_version` to the `resources` list in the `AgentgatewayPolicy` for your selected transport.


what is the main configuration?

Nadine2016 · 2026-09-03T17:29:07Z

+
+Agentgateway supports custom OpenTelemetry resource attributes through `spec.frontend.tracing.resources`. Resource attributes are added to every exported span and can help you filter and group traces in Arize AX.
+
+In Kubernetes mode, agentgateway automatically sets `service.name`, `service.version`, `service.instance.id`, and `service.namespace`. The main configuration explicitly sets `deployment.environment.name`. You can add application-specific attributes such as `model_id` or `model_version` to the `resources` list in the `AgentgatewayPolicy` for your selected transport.


Suggested change

In Kubernetes mode, agentgateway automatically sets `service.name`, `service.version`, `service.instance.id`, and `service.namespace`. The main configuration explicitly sets `deployment.environment.name`. You can add application-specific attributes such as `model_id` or `model_version` to the `resources` list in the `AgentgatewayPolicy` for your selected transport.

In Kubernetes mode, agentgateway automatically sets `service.name`, `service.version`, `service.instance.id`, and `service.namespace`. The main configuration explicitly sets `deployment.environment.name`. You can add application-specific attributes such as `model_id` or `model_version` to the `resources` list in the `{{< reuse "agw-docs/snippets/policy.md" >}}` for your selected transport.

Nadine2016 · 2026-09-03T17:29:43Z

+1. Verify that Kubernetes accepted the backend and attached the policy to the Gateway.
+   ```sh
+   kubectl get agentgatewaybackend arize-otlp -n agentgateway-system
+   kubectl get agentgatewaypolicy arize-tracing -n agentgateway-system


Suggested change

kubectl get agentgatewaypolicy arize-tracing -n agentgateway-system

kubectl get {{< reuse "agw-docs/snippets/policy.md" >}} arize-tracing -n {{< reuse "agw-docs/snippets/namespace.md" >}}

Nadine2016 · 2026-09-03T17:30:05Z

+
+1. Verify that Kubernetes accepted the backend and attached the policy to the Gateway.
+   ```sh
+   kubectl get agentgatewaybackend arize-otlp -n agentgateway-system


Suggested change

kubectl get agentgatewaybackend arize-otlp -n agentgateway-system

kubectl get agentgatewaybackend arize-otlp -n {{< reuse "agw-docs/snippets/namespace.md" >}}

Nadine2016 · 2026-09-03T17:30:27Z

+
+2. If you run a local cluster such as Kind, port-forward the agentgateway proxy.
+   ```sh
+   kubectl port-forward deployment/agentgateway-proxy -n agentgateway-system 8080:80


Suggested change

kubectl port-forward deployment/agentgateway-proxy -n agentgateway-system 8080:80

kubectl port-forward deployment/agentgateway-proxy -n {{< reuse "agw-docs/snippets/policy.md" >}} 8080:80

Nadine2016 · 2026-09-03T17:30:48Z

+   ```
+4. Find the request in the agentgateway proxy logs and copy its `trace.id` value.
+   ```sh
+   kubectl logs deployment/agentgateway-proxy -n agentgateway-system \


Suggested change

kubectl logs deployment/agentgateway-proxy -n agentgateway-system \

kubectl logs deployment/agentgateway-proxy -n {{< reuse "agw-docs/snippets/policy.md" >}} \

Nadine2016 · 2026-09-03T17:31:00Z

+   kubectl logs deployment/agentgateway-proxy -n agentgateway-system \
+     | grep 'protocol=llm' \
+     | tail -1
+   ```


can we provide a sample log entry?

Nadine2016 · 2026-09-03T17:34:31Z

+   ```sh
+   kubectl port-forward deployment/agentgateway-proxy -n agentgateway-system 8080:80
+   ```
+3. In a separate terminal, send an LLM request through agentgateway. The following example assumes that the proxy is available on local port `8080` and has an OpenAI-compatible provider and the `gpt-3.5-turbo` model configured.


we typically have Cloud Provider examples and local test examples in verification steps. https://docs.solo.io/agentgateway/kubernetes/latest/quickstart/non-agentic-http/#step-3-send-a-request-to-the-httpbin-app. I think just the curl command is enough as the before you begin section assumes you have the gateway up and running

Nadine2016 · 2026-09-03T17:34:51Z

+## Before you begin
+
+1. [Install agentgateway]({{< link-hextra path="/quickstart/install/" >}}) in your Kubernetes cluster.
+2. [Set up an agentgateway proxy]({{< link-hextra path="/setup/gateway/" >}}).


Get the external address of the gateway and save it in an environment variable.
{{< tabs >}}
{{% tab name="Cloud Provider LoadBalancer" %}}
{{< reuse "agw-docs/snippets/kind-loadbalancer-tip.md" >}}

export INGRESS_GW_ADDRESS=$(kubectl get svc -n {{< reuse "agw-docs/snippets/namespace.md" >}} agentgateway-proxy -o jsonpath="{.status.loadBalancer.ingress[0]['hostname','ip']}") echo $INGRESS_GW_ADDRESS

{{% /tab %}}
{{% tab name="Port-forward for local testing" %}}

kubectl port-forward deployment/agentgateway-proxy -n {{< reuse "agw-docs/snippets/namespace.md" >}} 8080:80

{{% /tab %}}
{{< /tabs >}}

Nadine2016 · 2026-09-03T17:35:21Z

+     | grep 'protocol=llm' \
+     | tail -1
+   ```
+5. In Arize AX, open **Tracing Projects**, select the project from `openinference.project.name`, and search for the trace ID. Trace export is batched, so allow several seconds for the trace to appear.


Suggested change

5. In Arize AX, open **Tracing Projects**, select the project from `openinference.project.name`, and search for the trace ID. Trace export is batched, so allow several seconds for the trace to appear.

5. In Arize AX, open **Tracing Projects**, select the project that you defined in the `openinference.project.name` field, and search for the trace ID. Trace export is batched, so allow several seconds for the trace to appear.

Nadine2016 · 2026-09-03T17:35:58Z

+- Set `randomSampling: "true"` while testing so that agentgateway starts a trace for every request.
+- Check the proxy logs for OpenTelemetry exporter errors.
+  ```sh
+  kubectl logs deployment/agentgateway-proxy -n agentgateway-system \


Suggested change

kubectl logs deployment/agentgateway-proxy -n agentgateway-system \

kubectl logs deployment/agentgateway-proxy -n {{< reuse "agw-docs/snippets/namespace.md" >}} \

Nadine2016 · 2026-09-03T17:36:55Z

@@ -0,0 +1,298 @@
+---
+title: Arize AX


can you move the content under assets/agw-docs/pages and then reference it from here and the page in main with a reuse shortcode?

docs: add Kubernetes Arize AX tracing guide

e30cd7d

Signed-off-by: Daneyon Hansen <daneyon.hansen@solo.io>

danehans mentioned this pull request Aug 28, 2026

docs: add observability platform integration guides agentgateway/agentgateway#3230

Open

28 tasks

docs: add main Arize AX Kubernetes guide

d5fbae3

Signed-off-by: Daneyon Hansen <daneyon.hansen@solo.io>

krisztianfekete reviewed Aug 31, 2026

View reviewed changes

danehans added 4 commits September 2, 2026 09:47

docs: address Arize tracing review

d4cbb0b

Signed-off-by: Daneyon Hansen <daneyon.hansen@solo.io>

Merge remote-tracking branch 'origin/main' into arize-ax-kubernetes-t…

aa321f4

…racing Signed-off-by: Daneyon Hansen <daneyon.hansen@solo.io>

docs: fix Arize guide CI checks

21e761c

Signed-off-by: Daneyon Hansen <daneyon.hansen@solo.io>

docs: update migrated observability links

ac75720

Signed-off-by: Daneyon Hansen <daneyon.hansen@solo.io>

danehans requested a review from krisztianfekete September 2, 2026 18:19

Nadine2016 reviewed Sep 3, 2026

View reviewed changes

	namespace: agentgateway-system
	namespace: {{< reuse "agw-docs/snippets/namespace.md" >}}

	- An `AgentgatewayPolicy` that exports sampled LLM traces from the `agentgateway-proxy` Gateway.
	- An `{{< reuse "agw-docs/snippets/policy.md" >}}` that exports sampled LLM traces from the `agentgateway-proxy` Gateway.

	kind: AgentgatewayPolicy
	kind: {{< reuse "agw-docs/snippets/policy.md" >}}

	apiVersion: agentgateway.dev/v1alpha1
	apiVersion: {{< reuse "agw-docs/snippets/api-version.md" >}}


		## Optional: Export LLM inputs and outputs

		The default configuration does not export prompt or response content. To display structured input and output messages in Arize AX, add the following attributes to the `AgentgatewayPolicy` for your selected transport.


		The default configuration does not export prompt or response content. To display structured input and output messages in Arize AX, add the following attributes to the `AgentgatewayPolicy` for your selected transport.

		{{< callout type="warning" >}}

	LLM prompts and responses can contain personally identifiable information (PII), credentials, or other sensitive data. Enabling these attributes sends that content to a third-party SaaS platform. Review your organization's data-handling requirements and configure appropriate guardrails or redaction before enabling them.
	> [!IMPORTANT]
	> LLM prompts and responses can contain personally identifiable information (PII), credentials, or other sensitive data. Enabling these attributes sends that content to a third-party SaaS platform. Review your organization's data-handling requirements and configure appropriate guardrails or redaction before enabling them.


		Agentgateway supports custom OpenTelemetry resource attributes through `spec.frontend.tracing.resources`. Resource attributes are added to every exported span and can help you filter and group traces in Arize AX.

		In Kubernetes mode, agentgateway automatically sets `service.name`, `service.version`, `service.instance.id`, and `service.namespace`. The main configuration explicitly sets `deployment.environment.name`. You can add application-specific attributes such as `model_id` or `model_version` to the `resources` list in the `AgentgatewayPolicy` for your selected transport.

	kubectl get agentgatewaypolicy arize-tracing -n agentgateway-system
	kubectl get {{< reuse "agw-docs/snippets/policy.md" >}} arize-tracing -n {{< reuse "agw-docs/snippets/namespace.md" >}}

	kubectl get agentgatewaybackend arize-otlp -n agentgateway-system
	kubectl get agentgatewaybackend arize-otlp -n {{< reuse "agw-docs/snippets/namespace.md" >}}

	kubectl port-forward deployment/agentgateway-proxy -n agentgateway-system 8080:80
	kubectl port-forward deployment/agentgateway-proxy -n {{< reuse "agw-docs/snippets/policy.md" >}} 8080:80

	kubectl logs deployment/agentgateway-proxy -n agentgateway-system \
	kubectl logs deployment/agentgateway-proxy -n {{< reuse "agw-docs/snippets/policy.md" >}} \

	5. In Arize AX, open Tracing Projects, select the project from `openinference.project.name`, and search for the trace ID. Trace export is batched, so allow several seconds for the trace to appear.
	5. In Arize AX, open Tracing Projects, select the project that you defined in the `openinference.project.name` field, and search for the trace ID. Trace export is batched, so allow several seconds for the trace to appear.

Uh oh!

Conversation

danehans commented Aug 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Company

Contribution summary

Checklist

Preview

Uh oh!

cloudflare-workers-and-pages Bot commented Aug 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Deploying agentproxy with Cloudflare Pages

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

danehans Sep 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

danehans Sep 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

danehans Sep 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Nadine2016 Sep 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

danehans commented Aug 28, 2026 •

edited

Loading

cloudflare-workers-and-pages Bot commented Aug 28, 2026 •

edited

Loading

danehans Sep 2, 2026 •

edited

Loading

danehans Sep 2, 2026 •

edited

Loading

danehans Sep 2, 2026 •

edited

Loading

Nadine2016 Sep 3, 2026 •

edited

Loading