diff --git a/assets/agw-docs/pages/about/overview.md b/assets/agw-docs/pages/about/overview.md index bf313ba76..d0db43052 100644 --- a/assets/agw-docs/pages/about/overview.md +++ b/assets/agw-docs/pages/about/overview.md @@ -83,7 +83,7 @@ The Gateway API is an API which can be used to program an ingress or an API gate {{< reuse "/agw-docs/snippets/agentgateway-capital.md" >}} provides the following extensions on top of the {{< reuse "agw-docs/snippets/k8s-gateway-api-name.md" >}} to configure advanced routing, security, and resiliency capabilities. {{< cards >}} - {{< card path="/security/access-logging/" title="Access logging" tag="Security" >}} + {{< card path="/observability/access-logs/view/" title="Access logging" tag="Security" >}} {{< card path="/integrations/aws-elb/" title="AWS ALB and NLB" tag="Traffic" >}} {{< card path="/traffic-management/destination-types/backends/lambda" title="AWS Lambda" tag="Traffic" >}} {{< card path="/traffic-management/route-delegation/" title="Delegation" tag="Traffic" >}} diff --git a/assets/agw-docs/pages/agentgateway/llm/cost-tracking.md b/assets/agw-docs/pages/agentgateway/llm/cost-tracking.md index 18889ad3b..0029d2fbd 100644 --- a/assets/agw-docs/pages/agentgateway/llm/cost-tracking.md +++ b/assets/agw-docs/pages/agentgateway/llm/cost-tracking.md @@ -169,7 +169,7 @@ groups: OpenTelemetry traces include token usage as span attributes. You can view per-request token counts in your tracing backend (such as Grafana Tempo, Jaeger, or Langfuse). -1. Set up OpenTelemetry tracing. See the [tracing guide]({{< link-hextra path="/observability/tracing/" >}}) for setup instructions. +1. Set up OpenTelemetry tracing. See the [tracing guide]({{< link-hextra path="/observability/traces/setup/" >}}) for setup instructions. 2. Search for traces with LLM requests. Each trace includes these attributes: - `gen_ai.usage.input_tokens`: Number of input tokens{{< version exclude-if="1.3.x,1.2.x,1.1.x,1.0.x,2.2.x" >}}, including the tokens read from or written to the prompt cache. For more information, see [Token usage fields]({{< link-hextra path="/llm/observability/#token-usage-fields" >}}).{{< /version >}} diff --git a/assets/agw-docs/pages/integrations/istio/ambient/ambient-ingress.md b/assets/agw-docs/pages/integrations/istio/ambient/ambient-ingress.md index f77193181..199df539b 100644 --- a/assets/agw-docs/pages/integrations/istio/ambient/ambient-ingress.md +++ b/assets/agw-docs/pages/integrations/istio/ambient/ambient-ingress.md @@ -272,7 +272,7 @@ Once the base flow works, layer additional agentgateway policies on the ingress Protects backends from spikes and abusive clients. 4. Traffic shaping and resilience, such as [traffic splitting]({{< link-hextra path="/traffic-management/traffic-split/" >}}), [retries]({{< link-hextra path="/resiliency/retry/retry/" >}}), and [request timeouts]({{< link-hextra path="/resiliency/timeouts/request/" >}}). Improves reliability of ingress traffic to ambient backends. -5. Observability, such as [access logs]({{< link-hextra path="/security/access-logging/" >}}) and [tracing]({{< link-hextra path="/observability/tracing/" >}}). +5. Observability, such as [access logs]({{< link-hextra path="/observability/access-logs/view/" >}}) and [tracing]({{< link-hextra path="/observability/traces/setup/" >}}). Gives per-request visibility into ingress traffic for compliance and incident response. ## Cleanup diff --git a/assets/agw-docs/pages/reference/api/latest/agentgateway.dev_agentgatewaypolicies.yaml.html b/assets/agw-docs/pages/reference/api/latest/agentgateway.dev_agentgatewaypolicies.yaml.html index dae1ca0f7..8ff3b9ece 100644 --- a/assets/agw-docs/pages/reference/api/latest/agentgateway.dev_agentgatewaypolicies.yaml.html +++ b/assets/agw-docs/pages/reference/api/latest/agentgateway.dev_agentgatewaypolicies.yaml.html @@ -2712,14 +2712,14 @@ - - - - - - + + + + + + - + @@ -2774,26 +2774,26 @@ - - - - - + + + + + - + - - - - + + + + - - - - - + + + + + diff --git a/assets/agw-docs/pages/reference/api/main/agentgateway.dev_agentgatewaypolicies.yaml.html b/assets/agw-docs/pages/reference/api/main/agentgateway.dev_agentgatewaypolicies.yaml.html index ebfa4818c..8b21d4ccc 100644 --- a/assets/agw-docs/pages/reference/api/main/agentgateway.dev_agentgatewaypolicies.yaml.html +++ b/assets/agw-docs/pages/reference/api/main/agentgateway.dev_agentgatewaypolicies.yaml.html @@ -2728,14 +2728,14 @@ - - - - - - + + + + + + - + @@ -2792,26 +2792,26 @@ - - - - - + + + + + - + - - - - + + + + - - - - - + + + + + diff --git a/assets/img/arize-ax-agentgateway-trace.png b/assets/img/arize-ax-agentgateway-trace.png new file mode 100644 index 000000000..6ff03b816 Binary files /dev/null and b/assets/img/arize-ax-agentgateway-trace.png differ diff --git a/content/docs/kubernetes/latest/integrations/llm-observability/_index.md b/content/docs/kubernetes/latest/integrations/llm-observability/_index.md index af1da6e1f..3e6eccbf5 100644 --- a/content/docs/kubernetes/latest/integrations/llm-observability/_index.md +++ b/content/docs/kubernetes/latest/integrations/llm-observability/_index.md @@ -1,7 +1,7 @@ --- title: LLM observability weight: 10 -description: Prompt logging, cost tracking, and audit trail via Langfuse, LangSmith, and more +description: Prompt logging, cost tracking, and audit trail via Arize AX, Langfuse, LangSmith, and more test: skip --- diff --git a/content/docs/kubernetes/latest/integrations/llm-observability/arize.md b/content/docs/kubernetes/latest/integrations/llm-observability/arize.md new file mode 100644 index 000000000..8a6ad9930 --- /dev/null +++ b/content/docs/kubernetes/latest/integrations/llm-observability/arize.md @@ -0,0 +1,298 @@ +--- +title: Arize AX +weight: 5 +description: Export agentgateway LLM traces to Arize AX over OTLP/HTTP or OTLP/gRPC. +test: skip +--- + +[Arize AX](https://arize.com/docs/ax) is an AI observability platform that accepts OpenTelemetry traces and displays LLM operations, models, and token usage. Agentgateway can export directly to Arize AX over OTLP/HTTP or OTLP/gRPC without a separate OpenTelemetry Collector. You can optionally export LLM inputs and outputs. + +## Before you begin + +1. [Install agentgateway]({{< link-hextra path="/quickstart/install/" >}}) in your Kubernetes cluster. +2. [Set up an agentgateway proxy]({{< link-hextra path="/setup/gateway/" >}}). +3. Set up an [LLM provider]({{< link-hextra path="/llm/providers/" >}}) and route in agentgateway. +4. **Arize account**: Sign up for an [Arize account](https://app.arize.com/auth/join). +5. **Arize API key and Space ID**: Obtain an API key and Space ID from the Arize platform. + +## Get your Arize API key and Space ID + +1. Log in to the [Arize dashboard](https://app.arize.com/). +2. Go to **Settings** > **API Keys** > **Service Keys**, and click **New Service Key**. +3. For **Account Role**, select **Member**. Add an organization, and then add the spaces that the service key can access. +4. Create the service key and copy the API service key, such as `ak-5245b124-1ef5-5514-...`. +5. Copy the base64 Space ID for the space that receives the traces, such as `U3BhY2U6TbN4WkU6wshdaf==`. The Space ID is different from the space name and organization ID. +6. Save the credentials in environment variables. Do not commit these values to source control. + ```sh + export ARIZE_API_KEY="" + export ARIZE_SPACE_ID="" + ``` +7. Create a Kubernetes Secret in the same namespace as the agentgateway proxy. + ```yaml + kubectl apply -f- <}} +{{% tab name="OTLP/HTTP" %}} + +For OTLP/HTTP, use the `arize-api-key` and `arize-space-id` headers. + +```yaml +kubectl apply -f- <<'EOF' +apiVersion: agentgateway.dev/v1alpha1 +kind: AgentgatewayBackend +metadata: + name: arize-otlp + namespace: agentgateway-system +spec: + static: + host: otlp.arize.com + port: 443 + policies: + tls: {} + auth: + credentials: + - location: + header: + name: arize-api-key + secretRef: + name: arize-credentials + key: api-key + - location: + header: + name: arize-space-id + secretRef: + name: arize-credentials + key: space-id +--- +apiVersion: agentgateway.dev/v1alpha1 +kind: AgentgatewayPolicy +metadata: + name: arize-tracing + namespace: agentgateway-system +spec: + targetRefs: + - group: gateway.networking.k8s.io + kind: Gateway + name: agentgateway-proxy + frontend: + tracing: + backendRef: + group: agentgateway.dev + kind: AgentgatewayBackend + name: arize-otlp + port: 443 + protocol: HTTP + randomSampling: "0.1" + clientSampling: "true" + filter: 'has(llm)' + resources: + - name: openinference.project.name + expression: '"agentgateway"' + - name: deployment.environment.name + expression: '"production"' +EOF +``` + +{{% /tab %}} +{{% tab name="OTLP/gRPC" %}} + +For OTLP/gRPC, use the `api_key` and `space_id` metadata headers. + +```yaml +kubectl apply -f- <<'EOF' +apiVersion: agentgateway.dev/v1alpha1 +kind: AgentgatewayBackend +metadata: + name: arize-otlp + namespace: agentgateway-system +spec: + static: + host: otlp.arize.com + port: 443 + policies: + tls: {} + auth: + credentials: + - location: + header: + name: api_key + secretRef: + name: arize-credentials + key: api-key + - location: + header: + name: space_id + secretRef: + name: arize-credentials + key: space-id +--- +apiVersion: agentgateway.dev/v1alpha1 +kind: AgentgatewayPolicy +metadata: + name: arize-tracing + namespace: agentgateway-system +spec: + targetRefs: + - group: gateway.networking.k8s.io + kind: Gateway + name: agentgateway-proxy + frontend: + tracing: + backendRef: + group: agentgateway.dev + kind: AgentgatewayBackend + name: arize-otlp + port: 443 + protocol: GRPC + randomSampling: "0.1" + clientSampling: "true" + filter: 'has(llm)' + resources: + - name: openinference.project.name + expression: '"agentgateway"' + - name: deployment.environment.name + expression: '"production"' +EOF +``` + +{{% /tab %}} +{{< /tabs >}} + +The values under `resources` are CEL expressions. The extra quotes around static values, such as `'"agentgateway"'`, make the CEL expression evaluate to a string. + +Change `openinference.project.name` if you want traces to appear in a different Arize project. Arize creates the project when it receives the first trace. + +Agentgateway emits model, provider, operation, and token usage attributes that follow the OpenTelemetry GenAI semantic conventions. Arize AX [natively maps these attributes](https://arize.com/blog/arize-ax-opentelemetry-genai-semantic-conventions/) to OpenInference fields. + +## Optional: Export LLM inputs and outputs + +The default configuration does not export prompt or response content. To display structured input and output messages in Arize AX, add the following attributes to the `AgentgatewayPolicy` for your selected transport. + +{{< callout type="warning" >}} +LLM prompts and responses can contain personally identifiable information (PII), credentials, or other sensitive data. Enabling these attributes sends that content to a third-party SaaS platform. Review your organization's data-handling requirements and configure appropriate guardrails or redaction before enabling them. +{{< /callout >}} + +```yaml +spec: + frontend: + tracing: + attributes: + add: + - name: llm.input_messages + expression: 'flattenRecursive(llm.prompt.map(c, {"message": c}))' + - name: llm.output_messages + expression: 'flattenRecursive(llm.completion.map(c, {"role": "assistant", "content": c}))' +``` + +## Optional: Add resource attributes + +Agentgateway supports custom OpenTelemetry resource attributes through `spec.frontend.tracing.resources`. Resource attributes are added to every exported span and can help you filter and group traces in Arize AX. + +In Kubernetes mode, agentgateway automatically sets `service.name`, `service.version`, `service.instance.id`, and `service.namespace`. The main configuration explicitly sets `deployment.environment.name`. You can add application-specific attributes such as `model_id` or `model_version` to the `resources` list in the `AgentgatewayPolicy` for your selected transport. + +```yaml +spec: + frontend: + tracing: + resources: + - name: openinference.project.name + expression: '"agentgateway"' + - name: deployment.environment.name + expression: '"production"' + - name: model_id + expression: '"gpt-4o-production"' + - name: model_version + expression: '"2026-08-27"' +``` + +Resource values are static CEL expressions that are initialized with the tracer and apply to every request. Do not set a static `service.instance.id`, which must identify a unique agentgateway replica. If agentgateway routes requests to multiple models, use the default `gen_ai.request.model` and `gen_ai.response.model` span attributes instead of setting a single `model_id` resource value. + +For more information, see [Add span and resource attributes]({{< link-hextra path="/observability/traces/setup/#add-attributes" >}}). + +## Verify the integration + +1. Verify that Kubernetes accepted the backend and attached the policy to the Gateway. + ```sh + kubectl get agentgatewaybackend arize-otlp -n agentgateway-system + kubectl get agentgatewaypolicy arize-tracing -n agentgateway-system + ``` + + Both resources should report `ACCEPTED=True`, and the policy should also report `ATTACHED=True`. + +2. If you run a local cluster such as Kind, port-forward the agentgateway proxy. + ```sh + kubectl port-forward deployment/agentgateway-proxy -n agentgateway-system 8080:80 + ``` +3. In a separate terminal, send an LLM request through agentgateway. The following example assumes that the proxy is available on local port `8080` and has an OpenAI-compatible provider and the `gpt-3.5-turbo` model configured. + ```sh + curl http://localhost:8080/v1/chat/completions \ + -H 'content-type: application/json' \ + -d '{ + "model": "gpt-3.5-turbo", + "messages": [ + { + "role": "user", + "content": "Reply with exactly: Arize tracing works" + } + ] + }' + ``` +4. Find the request in the agentgateway proxy logs and copy its `trace.id` value. + ```sh + kubectl logs deployment/agentgateway-proxy -n agentgateway-system \ + | grep 'protocol=llm' \ + | tail -1 + ``` +5. In Arize AX, open **Tracing Projects**, select the project from `openinference.project.name`, and search for the trace ID. Trace export is batched, so allow several seconds for the trace to appear. + +{{< reuse-image src="img/arize-ax-agentgateway-trace.png" srcDark="img/arize-ax-agentgateway-trace.png" alt="Arize AX showing an agentgateway openai.chat trace with its input, output, latency, cost, and token count" caption="An agentgateway LLM trace in Arize AX." >}} + +## Troubleshoot trace export + +- Use the header names that correspond to your selected protocol: hyphenated headers for OTLP/HTTP and underscore headers for OTLP/gRPC. +- Confirm that the `arize-credentials` Secret is in the same namespace as the `arize-otlp` backend and contains the `api-key` and `space-id` keys. +- Confirm that the collector host matches your Arize region and that `openinference.project.name` is set. +- Check the `ACCEPTED` and `ATTACHED` status columns for the backend and policy. +- Set `randomSampling: "true"` while testing so that agentgateway starts a trace for every request. +- Check the proxy logs for OpenTelemetry exporter errors. + ```sh + kubectl logs deployment/agentgateway-proxy -n agentgateway-system \ + | grep -i opentelemetry + ``` + +For more information about Arize authentication and OpenTelemetry export, see the [Arize AX manual instrumentation documentation](https://arize.com/docs/ax/instrument/manual-instrumentation). diff --git a/content/docs/kubernetes/latest/observability/otel-stack.md b/content/docs/kubernetes/latest/observability/otel-stack.md index b05a2c689..f8009a7d3 100644 --- a/content/docs/kubernetes/latest/observability/otel-stack.md +++ b/content/docs/kubernetes/latest/observability/otel-stack.md @@ -56,13 +56,13 @@ EOF 1. Open and log in to Grafana with the username `admin` and password `prom-operator`. - {{< tabs tabTotal="2" items="Cloud Provider LoadBalancer,Port-forward for local testing" >}} - {{% tab tabName="Cloud Provider LoadBalancer" %}} + {{< tabs >}} + {{% tab name="Cloud Provider LoadBalancer" %}} ```sh open "http://$(kubectl -n telemetry get svc kube-prometheus-stack-grafana -o jsonpath="{.status.loadBalancer.ingress[0]['hostname','ip']}"):3000" ``` {{% /tab %}} - {{% tab tabName="Port-forward for local testing" %}} + {{% tab name="Port-forward for local testing" %}} 1. Port-forward the Grafana service to your local machine. ```sh kubectl port-forward deployment/kube-prometheus-stack-grafana -n telemetry 3000 @@ -136,4 +136,4 @@ The OTel stack provides the storage and visualization layer. Use the following g 4. Remove the `telemetry` namespace. ```sh kubectl delete namespace telemetry - ``` \ No newline at end of file + ``` diff --git a/content/docs/kubernetes/main/integrations/llm-observability/_index.md b/content/docs/kubernetes/main/integrations/llm-observability/_index.md index af1da6e1f..3e6eccbf5 100644 --- a/content/docs/kubernetes/main/integrations/llm-observability/_index.md +++ b/content/docs/kubernetes/main/integrations/llm-observability/_index.md @@ -1,7 +1,7 @@ --- title: LLM observability weight: 10 -description: Prompt logging, cost tracking, and audit trail via Langfuse, LangSmith, and more +description: Prompt logging, cost tracking, and audit trail via Arize AX, Langfuse, LangSmith, and more test: skip --- diff --git a/content/docs/kubernetes/main/integrations/llm-observability/arize.md b/content/docs/kubernetes/main/integrations/llm-observability/arize.md new file mode 100644 index 000000000..8a6ad9930 --- /dev/null +++ b/content/docs/kubernetes/main/integrations/llm-observability/arize.md @@ -0,0 +1,298 @@ +--- +title: Arize AX +weight: 5 +description: Export agentgateway LLM traces to Arize AX over OTLP/HTTP or OTLP/gRPC. +test: skip +--- + +[Arize AX](https://arize.com/docs/ax) is an AI observability platform that accepts OpenTelemetry traces and displays LLM operations, models, and token usage. Agentgateway can export directly to Arize AX over OTLP/HTTP or OTLP/gRPC without a separate OpenTelemetry Collector. You can optionally export LLM inputs and outputs. + +## Before you begin + +1. [Install agentgateway]({{< link-hextra path="/quickstart/install/" >}}) in your Kubernetes cluster. +2. [Set up an agentgateway proxy]({{< link-hextra path="/setup/gateway/" >}}). +3. Set up an [LLM provider]({{< link-hextra path="/llm/providers/" >}}) and route in agentgateway. +4. **Arize account**: Sign up for an [Arize account](https://app.arize.com/auth/join). +5. **Arize API key and Space ID**: Obtain an API key and Space ID from the Arize platform. + +## Get your Arize API key and Space ID + +1. Log in to the [Arize dashboard](https://app.arize.com/). +2. Go to **Settings** > **API Keys** > **Service Keys**, and click **New Service Key**. +3. For **Account Role**, select **Member**. Add an organization, and then add the spaces that the service key can access. +4. Create the service key and copy the API service key, such as `ak-5245b124-1ef5-5514-...`. +5. Copy the base64 Space ID for the space that receives the traces, such as `U3BhY2U6TbN4WkU6wshdaf==`. The Space ID is different from the space name and organization ID. +6. Save the credentials in environment variables. Do not commit these values to source control. + ```sh + export ARIZE_API_KEY="" + export ARIZE_SPACE_ID="" + ``` +7. Create a Kubernetes Secret in the same namespace as the agentgateway proxy. + ```yaml + kubectl apply -f- <}} +{{% tab name="OTLP/HTTP" %}} + +For OTLP/HTTP, use the `arize-api-key` and `arize-space-id` headers. + +```yaml +kubectl apply -f- <<'EOF' +apiVersion: agentgateway.dev/v1alpha1 +kind: AgentgatewayBackend +metadata: + name: arize-otlp + namespace: agentgateway-system +spec: + static: + host: otlp.arize.com + port: 443 + policies: + tls: {} + auth: + credentials: + - location: + header: + name: arize-api-key + secretRef: + name: arize-credentials + key: api-key + - location: + header: + name: arize-space-id + secretRef: + name: arize-credentials + key: space-id +--- +apiVersion: agentgateway.dev/v1alpha1 +kind: AgentgatewayPolicy +metadata: + name: arize-tracing + namespace: agentgateway-system +spec: + targetRefs: + - group: gateway.networking.k8s.io + kind: Gateway + name: agentgateway-proxy + frontend: + tracing: + backendRef: + group: agentgateway.dev + kind: AgentgatewayBackend + name: arize-otlp + port: 443 + protocol: HTTP + randomSampling: "0.1" + clientSampling: "true" + filter: 'has(llm)' + resources: + - name: openinference.project.name + expression: '"agentgateway"' + - name: deployment.environment.name + expression: '"production"' +EOF +``` + +{{% /tab %}} +{{% tab name="OTLP/gRPC" %}} + +For OTLP/gRPC, use the `api_key` and `space_id` metadata headers. + +```yaml +kubectl apply -f- <<'EOF' +apiVersion: agentgateway.dev/v1alpha1 +kind: AgentgatewayBackend +metadata: + name: arize-otlp + namespace: agentgateway-system +spec: + static: + host: otlp.arize.com + port: 443 + policies: + tls: {} + auth: + credentials: + - location: + header: + name: api_key + secretRef: + name: arize-credentials + key: api-key + - location: + header: + name: space_id + secretRef: + name: arize-credentials + key: space-id +--- +apiVersion: agentgateway.dev/v1alpha1 +kind: AgentgatewayPolicy +metadata: + name: arize-tracing + namespace: agentgateway-system +spec: + targetRefs: + - group: gateway.networking.k8s.io + kind: Gateway + name: agentgateway-proxy + frontend: + tracing: + backendRef: + group: agentgateway.dev + kind: AgentgatewayBackend + name: arize-otlp + port: 443 + protocol: GRPC + randomSampling: "0.1" + clientSampling: "true" + filter: 'has(llm)' + resources: + - name: openinference.project.name + expression: '"agentgateway"' + - name: deployment.environment.name + expression: '"production"' +EOF +``` + +{{% /tab %}} +{{< /tabs >}} + +The values under `resources` are CEL expressions. The extra quotes around static values, such as `'"agentgateway"'`, make the CEL expression evaluate to a string. + +Change `openinference.project.name` if you want traces to appear in a different Arize project. Arize creates the project when it receives the first trace. + +Agentgateway emits model, provider, operation, and token usage attributes that follow the OpenTelemetry GenAI semantic conventions. Arize AX [natively maps these attributes](https://arize.com/blog/arize-ax-opentelemetry-genai-semantic-conventions/) to OpenInference fields. + +## Optional: Export LLM inputs and outputs + +The default configuration does not export prompt or response content. To display structured input and output messages in Arize AX, add the following attributes to the `AgentgatewayPolicy` for your selected transport. + +{{< callout type="warning" >}} +LLM prompts and responses can contain personally identifiable information (PII), credentials, or other sensitive data. Enabling these attributes sends that content to a third-party SaaS platform. Review your organization's data-handling requirements and configure appropriate guardrails or redaction before enabling them. +{{< /callout >}} + +```yaml +spec: + frontend: + tracing: + attributes: + add: + - name: llm.input_messages + expression: 'flattenRecursive(llm.prompt.map(c, {"message": c}))' + - name: llm.output_messages + expression: 'flattenRecursive(llm.completion.map(c, {"role": "assistant", "content": c}))' +``` + +## Optional: Add resource attributes + +Agentgateway supports custom OpenTelemetry resource attributes through `spec.frontend.tracing.resources`. Resource attributes are added to every exported span and can help you filter and group traces in Arize AX. + +In Kubernetes mode, agentgateway automatically sets `service.name`, `service.version`, `service.instance.id`, and `service.namespace`. The main configuration explicitly sets `deployment.environment.name`. You can add application-specific attributes such as `model_id` or `model_version` to the `resources` list in the `AgentgatewayPolicy` for your selected transport. + +```yaml +spec: + frontend: + tracing: + resources: + - name: openinference.project.name + expression: '"agentgateway"' + - name: deployment.environment.name + expression: '"production"' + - name: model_id + expression: '"gpt-4o-production"' + - name: model_version + expression: '"2026-08-27"' +``` + +Resource values are static CEL expressions that are initialized with the tracer and apply to every request. Do not set a static `service.instance.id`, which must identify a unique agentgateway replica. If agentgateway routes requests to multiple models, use the default `gen_ai.request.model` and `gen_ai.response.model` span attributes instead of setting a single `model_id` resource value. + +For more information, see [Add span and resource attributes]({{< link-hextra path="/observability/traces/setup/#add-attributes" >}}). + +## Verify the integration + +1. Verify that Kubernetes accepted the backend and attached the policy to the Gateway. + ```sh + kubectl get agentgatewaybackend arize-otlp -n agentgateway-system + kubectl get agentgatewaypolicy arize-tracing -n agentgateway-system + ``` + + Both resources should report `ACCEPTED=True`, and the policy should also report `ATTACHED=True`. + +2. If you run a local cluster such as Kind, port-forward the agentgateway proxy. + ```sh + kubectl port-forward deployment/agentgateway-proxy -n agentgateway-system 8080:80 + ``` +3. In a separate terminal, send an LLM request through agentgateway. The following example assumes that the proxy is available on local port `8080` and has an OpenAI-compatible provider and the `gpt-3.5-turbo` model configured. + ```sh + curl http://localhost:8080/v1/chat/completions \ + -H 'content-type: application/json' \ + -d '{ + "model": "gpt-3.5-turbo", + "messages": [ + { + "role": "user", + "content": "Reply with exactly: Arize tracing works" + } + ] + }' + ``` +4. Find the request in the agentgateway proxy logs and copy its `trace.id` value. + ```sh + kubectl logs deployment/agentgateway-proxy -n agentgateway-system \ + | grep 'protocol=llm' \ + | tail -1 + ``` +5. In Arize AX, open **Tracing Projects**, select the project from `openinference.project.name`, and search for the trace ID. Trace export is batched, so allow several seconds for the trace to appear. + +{{< reuse-image src="img/arize-ax-agentgateway-trace.png" srcDark="img/arize-ax-agentgateway-trace.png" alt="Arize AX showing an agentgateway openai.chat trace with its input, output, latency, cost, and token count" caption="An agentgateway LLM trace in Arize AX." >}} + +## Troubleshoot trace export + +- Use the header names that correspond to your selected protocol: hyphenated headers for OTLP/HTTP and underscore headers for OTLP/gRPC. +- Confirm that the `arize-credentials` Secret is in the same namespace as the `arize-otlp` backend and contains the `api-key` and `space-id` keys. +- Confirm that the collector host matches your Arize region and that `openinference.project.name` is set. +- Check the `ACCEPTED` and `ATTACHED` status columns for the backend and policy. +- Set `randomSampling: "true"` while testing so that agentgateway starts a trace for every request. +- Check the proxy logs for OpenTelemetry exporter errors. + ```sh + kubectl logs deployment/agentgateway-proxy -n agentgateway-system \ + | grep -i opentelemetry + ``` + +For more information about Arize authentication and OpenTelemetry export, see the [Arize AX manual instrumentation documentation](https://arize.com/docs/ax/instrument/manual-instrumentation). diff --git a/content/docs/kubernetes/main/observability/otel-stack.md b/content/docs/kubernetes/main/observability/otel-stack.md index b05a2c689..f8009a7d3 100644 --- a/content/docs/kubernetes/main/observability/otel-stack.md +++ b/content/docs/kubernetes/main/observability/otel-stack.md @@ -56,13 +56,13 @@ EOF 1. Open and log in to Grafana with the username `admin` and password `prom-operator`. - {{< tabs tabTotal="2" items="Cloud Provider LoadBalancer,Port-forward for local testing" >}} - {{% tab tabName="Cloud Provider LoadBalancer" %}} + {{< tabs >}} + {{% tab name="Cloud Provider LoadBalancer" %}} ```sh open "http://$(kubectl -n telemetry get svc kube-prometheus-stack-grafana -o jsonpath="{.status.loadBalancer.ingress[0]['hostname','ip']}"):3000" ``` {{% /tab %}} - {{% tab tabName="Port-forward for local testing" %}} + {{% tab name="Port-forward for local testing" %}} 1. Port-forward the Grafana service to your local machine. ```sh kubectl port-forward deployment/kube-prometheus-stack-grafana -n telemetry 3000 @@ -136,4 +136,4 @@ The OTel stack provides the storage and visualization layer. Use the following g 4. Remove the `telemetry` namespace. ```sh kubectl delete namespace telemetry - ``` \ No newline at end of file + ```