Skip to content

generators: make the ollama empty-response retry actually retry - #2122

Open
VaggelisGian wants to merge 1 commit into
NVIDIA:mainfrom
VaggelisGian:fix-ollama-empty-response-retry
Open

generators: make the ollama empty-response retry actually retry#2122
VaggelisGian wants to merge 1 commit into
NVIDIA:mainfrom
VaggelisGian:fix-ollama-empty-response-retry

Conversation

@VaggelisGian

Copy link
Copy Markdown

What this change does

The backoff.on_predicate guard on both ollama _call_model variants tested ans == [None] or len(ans) == 0, but the wrapped functions only ever return a one-element list holding a Message, so neither condition could ever be true. The retry was dead code: an Ollama generation with no content ({"response": null}, or a chat response missing content) surfaced as Message(text=None) on the first attempt, every time.

The predicate now retries when the list is empty, its first element is None, or the message carries no text, which is what the decorator and its comment always intended. Bounds are unchanged:

  • content-bearing generations still take exactly one attempt (pinned by test);
  • responses that keep coming back empty still surface unchanged after max_tries=3, so downstream consumers see exactly what they saw before;
  • worst case added latency for a persistently empty generation is about one second of backoff sleep plus two extra HTTP calls, per call.

Both generator classes (OllamaGenerator, OllamaGeneratorChat) get the same predicate.

Why this is not a duplicate

Searches for open or recent PRs touching this returned nothing overlapping:

  • gh pr list --repo NVIDIA/garak --state all --search "ollama" shows only unrelated feature work (image scaling probes, auth, options forwarding) and nothing about the empty-response retry
  • the retry predicate dates back to the original ollama generator and was last touched by refactors that moved it verbatim

Verification

Environment: garak 0.16.1.pre1 main, Python 3.12.6, Windows 11 Pro.

New tests fail before the fix, pass after (watched both). Before the fix both retry tests fail because only one HTTP request happens; after the fix the second request returns the recovered text:

python -m pytest tests/generators/test_ollama.py -q
  -> 13 passed, 4 skipped in 9.57s   (skips need a live Ollama server)

python -m pytest tests/generators/test_generators.py -q
  -> 117 passed in 29.27s

python -m black --config pyproject.toml --check garak/generators/ollama.py tests/generators/test_ollama.py
  -> 2 files would be left unchanged

Direct behavioral proof with a fake client whose generate() always returns an empty payload:

EMPTY CASE:
calls: 3            <- max_tries honoured
returned text: None (Message, unchanged after exhaustion)
CONTENT CASE:
calls: 1            <- no retry when there is content

Checklist notes:

  • Supporting configuration such as generator configuration file: none needed, no params changed
  • garak -t <target_type> -n <model_name>: not run end to end; needs a live Ollama server which is not available here. Coverage comes from the mocked respx suite plus the direct behavioral proof above.
  • Run the tests: commands and literal results above
  • Verify the thing does what it should: empty generation retried up to max_tries, recovered response surfaced
  • Verify the thing does not do what it should not: content-bearing generation takes exactly one attempt; exhausted retries return the same value as before
  • Documentation: no new plugin class or module; behavior now matches the existing decorator comment

Statement on AI assistance

AI assistance was used for this contribution: an AI coding agent drafted the change and tests under my direction; I reviewed every changed line, ran all quoted commands myself, and submit this PR as its author. It fixes a self-found defect, verified against a live repro before this PR, and is not a duplicate per the section above.

The on_predicate guard on both _call_model variants tested
ans == [None] or len(ans) == 0, but the wrapped functions only ever
return a one-element list holding a Message, so neither condition could
be true and empty responses were never retried: an Ollama generation
with no content surfaced as Message(None) on the first attempt.

The predicate now retries when the list is empty, its element is None,
or the message carries no text, matching the stated intent of the
decorator. Responses that keep coming back empty still surface
unchanged after max_tries, so downstream consumers see what they saw
before; content-bearing generations still take exactly one attempt.

Test Plan:
  python -m pytest tests/generators/test_ollama.py -q
    -> 13 passed, 4 skipped (server-dependent)
  python -m pytest tests/generators/test_generators.py -q
    -> 117 passed
  python -m black --config pyproject.toml --check     garak/generators/ollama.py tests/generators/test_ollama.py
    -> 2 files would be left unchanged

Signed-off-by: Vaggelis <baggelis100@gmail.com>
@jmartin-tech jmartin-tech self-assigned this Aug 29, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants