Repository navigation
fix(litellm): end spans on failed calls, unread streams, and non-completed responses - #3948
Open
sushantsimpplr wants to merge 1 commit into
Open
sushantsimpplr wants to merge 1 commit into
sushantsimpplr wants to merge 1 commit into
Conversation
…leted responses The streaming branches started a span and awaited litellm outside any try/except, so a failed or cancelled call never ended its span. A returned stream that was discarded or closed before its first read never ran its finalizer either. Responses streaming finalizers ended the span only on response.completed, leaving response.incomplete, response.failed and mid-stream errors with an open span. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
chrikrah
approved these changes
Oct 11, 2026
chrikrah
left a comment
There was a problem hiding this comment.
@sushantsimpplr approving at 2329c35. All eight new tests fail with the base __init__.py and pass here.
$ python -m pytest tests -q -p no:cacheprovider # openinference-instrumentation-litellm, litellm 1.84.0 (test-requirements.txt), Python 3.12
# head 2329c35
134 passed
# head tests, __init__.py from base 19363a0
8 failed, 126 passed # every new case in test_streaming_transparency.py
non-blocking: cancellation now splits by stream. The stream span ends ERROR, matching #3961 and the anthropic _stream.py, while non-stream acompletion keeps the UNSET it had at base, because use_span only catches Exception. With original_litellm_funcs["acompletion"] swapped for a raising stub:
$ python -m pytest cancel_probe.py -q -s # head 2329c35
PROBE stream=False RuntimeError spans=1 status=ERROR events=['exception']
PROBE stream=False CancelledError spans=1 status=UNSET events=[]
PROBE stream=True RuntimeError spans=1 status=ERROR events=['exception']
PROBE stream=True CancelledError spans=1 status=ERROR events=['exception']
# same probe, __init__.py from base 19363a0
PROBE stream=False RuntimeError spans=1 status=ERROR events=['exception']
PROBE stream=False CancelledError spans=1 status=UNSET events=[]
PROBE stream=True RuntimeError spans=0 status=- events=-
PROBE stream=True CancelledError spans=0 status=- events=-
non-blocking: chat-completion streams that fail mid-stream still end UNSET here, while the responses finalizers now set ERROR. #4015 adds that status, and both it and #4016 merge cleanly with this branch (git merge-tree).
@caroger, should the non-stream branch follow in this PR or in a separate one?
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Follow-up to #3578. Several paths leave a litellm span open forever. An open span is never exported, and any span processor that tracks live spans holds it for the life of the process. We hit this as steady memory growth in a LiteLLM proxy deployment.
Changes
completion,acompletion,responsesandaresponsesstreaming branches start a span and then call litellm outside anytry. A provider error or a cancellation now records the exception, setsERRORstatus and ends the span before re-raising. The non-streaming branches already get this fromstart_as_current_span.finally, so a returned stream discarded or closed before its first read never ended its span._TracedSyncStream/_TracedAsyncStreamnow track whether iteration started, and end the span onclose()/aclose()or on collection if it did not. Once iteration starts, the finalizer owns the span as before.response.completed. They now also finalize onresponse.incompleteandresponse.failed, and end the span (recording the exception) when the stream errors or is closed before a terminal event._finalize_spanalready setsERRORfrom a failed response'serror.Testing
tests/test_streaming_transparency.py: failed call (provider error andCancelledError), async and sync streams discarded unread, async stream closed unread (span ends once;completion_streamstill released),response.incomplete/response.failed, and a mid-stream error. All 8 fail onmainand pass with this change.test-requirements.txt); the new tests also pass on litellm 1.94.1.ruff0.9.2 andmypy1.11.2 (strict,mypy .as in tox) clean.🤖 Generated with Claude Code