generators: make the ollama empty-response retry actually retry - #2122
Open
VaggelisGian wants to merge 1 commit into
Open
generators: make the ollama empty-response retry actually retry#2122VaggelisGian wants to merge 1 commit into
VaggelisGian wants to merge 1 commit into
Conversation
The on_predicate guard on both _call_model variants tested
ans == [None] or len(ans) == 0, but the wrapped functions only ever
return a one-element list holding a Message, so neither condition could
be true and empty responses were never retried: an Ollama generation
with no content surfaced as Message(None) on the first attempt.
The predicate now retries when the list is empty, its element is None,
or the message carries no text, matching the stated intent of the
decorator. Responses that keep coming back empty still surface
unchanged after max_tries, so downstream consumers see what they saw
before; content-bearing generations still take exactly one attempt.
Test Plan:
python -m pytest tests/generators/test_ollama.py -q
-> 13 passed, 4 skipped (server-dependent)
python -m pytest tests/generators/test_generators.py -q
-> 117 passed
python -m black --config pyproject.toml --check garak/generators/ollama.py tests/generators/test_ollama.py
-> 2 files would be left unchanged
Signed-off-by: Vaggelis <baggelis100@gmail.com>
jmartin-tech
approved these changes
Aug 29, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this change does
The
backoff.on_predicateguard on both ollama_call_modelvariants testedans == [None] or len(ans) == 0, but the wrapped functions only ever return a one-element list holding aMessage, so neither condition could ever be true. The retry was dead code: an Ollama generation with no content ({"response": null}, or a chat response missingcontent) surfaced asMessage(text=None)on the first attempt, every time.The predicate now retries when the list is empty, its first element is None, or the message carries no text, which is what the decorator and its comment always intended. Bounds are unchanged:
max_tries=3, so downstream consumers see exactly what they saw before;Both generator classes (
OllamaGenerator,OllamaGeneratorChat) get the same predicate.Why this is not a duplicate
Searches for open or recent PRs touching this returned nothing overlapping:
gh pr list --repo NVIDIA/garak --state all --search "ollama"shows only unrelated feature work (image scaling probes, auth, options forwarding) and nothing about the empty-response retryVerification
Environment: garak 0.16.1.pre1 main, Python 3.12.6, Windows 11 Pro.
New tests fail before the fix, pass after (watched both). Before the fix both retry tests fail because only one HTTP request happens; after the fix the second request returns the recovered text:
Direct behavioral proof with a fake client whose
generate()always returns an empty payload:Checklist notes:
garak -t <target_type> -n <model_name>: not run end to end; needs a live Ollama server which is not available here. Coverage comes from the mocked respx suite plus the direct behavioral proof above.Statement on AI assistance
AI assistance was used for this contribution: an AI coding agent drafted the change and tests under my direction; I reviewed every changed line, ran all quoted commands myself, and submit this PR as its author. It fixes a self-found defect, verified against a live repro before this PR, and is not a duplicate per the section above.