Skip to content

feat(observability): add GenAI metrics for Prompt Drivers - #2368

Open
kskszq wants to merge 4 commits into
griptape-ai:mainfrom
kskszq:feat/prompt-driver-metrics-1790
Open

kskszq wants to merge 4 commits into
griptape-ai:mainfrom
kskszq:feat/prompt-driver-metrics-1790

Conversation

@kskszq

@kskszq kskszq commented Oct 3, 2026 •

Copy link
Copy Markdown

Describe your changes

OpenTelemetry observability currently exports traces but does not export GenAI metrics for Prompt Drivers. Add an optional SDK meter_provider to OpenTelemetryObservabilityDriver, falling back to OpenTelemetry's global meter provider, and record metrics around PromptDriver.run() while preserving existing tracing behavior.

The implementation records operation duration, input/output token usage counters, and per-operation input/output token histograms using the current GenAI metric conventions and inference token conventions. Metrics include provider, operation, and requested model; failures include the exception class as error.type. Aggregate token counters use the unknown modality. Prompt and response content are not included in metric attributes.

Duration covers one logical Driver run, including retries. Token usage is recorded only from its final successful response; streaming uses the existing final cumulative usage aggregation. Unknown token counts are omitted, and reported zero counts are retained. Failed retry usage is unavailable and is not estimated. The supplied provider is flushed, but not shut down, when the context exits. Streaming time-to-first-chunk and per-chunk latency are outside this change.

Include a console metrics example, configuration/lifecycle documentation, and regression coverage for success, streaming, missing/zero usage, errors, retries, provider subclasses, global providers, context reuse, and calls outside Prompt Drivers.

Validation on Python 3.13.7 with the repository's frozen core/dev/OpenTelemetry dependencies:

  • python -m pytest tests/unit/drivers/observability tests/unit/drivers/prompt/test_base_prompt_driver.py tests/unit/observability tests/unit/common/test_observable.py tests/unit/structures tests/unit/tasks/test_prompt_task.py -q: 233 passed.
  • ruff check and ruff format --check: passed across the repository.
  • pyright on the changed Driver and new documentation example: 0 errors.
  • mdformat --check .github/ docs/: passed after normalizing Windows checkout line endings for the check; unrelated files retain their original checkout contents.
  • typos and git diff --cached --check: passed.

The complete unit/integration suite and live provider/exporter integrations were not run. Tests use in-memory metrics and mock Prompt Drivers; no model API calls are needed.

Issue ticket number and link

Addresses #1790: #1790


📚 Documentation preview 📚: https://griptape--2368.org.readthedocs.build//2368/

Copilot AI balanced review requested due to automatic review settings October 3, 2026 15:54

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@codecov

codecov Bot commented Oct 3, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants