Skip to content

fix(litellm): end spans on failed calls, unread streams, and non-completed responses - #3948

Open
sushantsimpplr wants to merge 1 commit into
Arize-ai:mainfrom
sushantsimpplr:fix/litellm-span-lifecycle
Open

sushantsimpplr wants to merge 1 commit into
Arize-ai:mainfrom
sushantsimpplr:fix/litellm-span-lifecycle

Conversation

@sushantsimpplr

Copy link
Copy Markdown
Contributor

Description

Follow-up to #3578. Several paths leave a litellm span open forever. An open span is never exported, and any span processor that tracks live spans holds it for the life of the process. We hit this as steady memory growth in a LiteLLM proxy deployment.

Changes

  1. Failed or cancelled streaming calls. The completion, acompletion, responses and aresponses streaming branches start a span and then call litellm outside any try. A provider error or a cancellation now records the exception, sets ERROR status and ends the span before re-raising. The non-streaming branches already get this from start_as_current_span.
  2. Streams never read. A generator that never starts never runs its finally, so a returned stream discarded or closed before its first read never ended its span. _TracedSyncStream / _TracedAsyncStream now track whether iteration started, and end the span on close() / aclose() or on collection if it did not. Once iteration starts, the finalizer owns the span as before.
  3. Responses terminal events. The Responses streaming finalizers ended the span only on response.completed. They now also finalize on response.incomplete and response.failed, and end the span (recording the exception) when the stream errors or is closed before a terminal event. _finalize_span already sets ERROR from a failed response's error.

Testing

  • 8 new tests in tests/test_streaming_transparency.py: failed call (provider error and CancelledError), async and sync streams discarded unread, async stream closed unread (span ends once; completion_stream still released), response.incomplete / response.failed, and a mid-stream error. All 8 fail on main and pass with this change.
  • Full package suite: 134 passed on litellm 1.84.0 (test-requirements.txt); the new tests also pass on litellm 1.94.1.
  • ruff 0.9.2 and mypy 1.11.2 (strict, mypy . as in tox) clean.

🤖 Generated with Claude Code

…leted responses

The streaming branches started a span and awaited litellm outside any
try/except, so a failed or cancelled call never ended its span. A returned
stream that was discarded or closed before its first read never ran its
finalizer either. Responses streaming finalizers ended the span only on
response.completed, leaving response.incomplete, response.failed and
mid-stream errors with an open span.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@chrikrah chrikrah left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@sushantsimpplr approving at 2329c35. All eight new tests fail with the base __init__.py and pass here.

$ python -m pytest tests -q -p no:cacheprovider    # openinference-instrumentation-litellm, litellm 1.84.0 (test-requirements.txt), Python 3.12
# head 2329c35
134 passed
# head tests, __init__.py from base 19363a0
8 failed, 126 passed    # every new case in test_streaming_transparency.py

non-blocking: cancellation now splits by stream. The stream span ends ERROR, matching #3961 and the anthropic _stream.py, while non-stream acompletion keeps the UNSET it had at base, because use_span only catches Exception. With original_litellm_funcs["acompletion"] swapped for a raising stub:

$ python -m pytest cancel_probe.py -q -s    # head 2329c35
PROBE stream=False RuntimeError   spans=1 status=ERROR events=['exception']
PROBE stream=False CancelledError spans=1 status=UNSET events=[]
PROBE stream=True  RuntimeError   spans=1 status=ERROR events=['exception']
PROBE stream=True  CancelledError spans=1 status=ERROR events=['exception']
# same probe, __init__.py from base 19363a0
PROBE stream=False RuntimeError   spans=1 status=ERROR events=['exception']
PROBE stream=False CancelledError spans=1 status=UNSET events=[]
PROBE stream=True  RuntimeError   spans=0 status=- events=-
PROBE stream=True  CancelledError spans=0 status=- events=-

non-blocking: chat-completion streams that fail mid-stream still end UNSET here, while the responses finalizers now set ERROR. #4015 adds that status, and both it and #4016 merge cleanly with this branch (git merge-tree).

@caroger, should the non-stream branch follow in this PR or in a separate one?

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: No status

Development

Successfully merging this pull request may close these issues.

2 participants