fix(generators): raise clear error on HF InferenceAPI 404 response - #2112
Open
pujitha24 wants to merge 1 commit into
Open
fix(generators): raise clear error on HF InferenceAPI 404 response#2112pujitha24 wants to merge 1 commit into
pujitha24 wants to merge 1 commit into
Conversation
Motivation: huggingface.InferenceAPI raised a confusing generic
TypeError ("Unsure how to parse API response type... please open an
issue") when the Hugging Face Inference API returned an HTTP 404 for a
model. This is exactly what happened in the reported issue: the model
is not exposed via the legacy default Inference API provider (Hugging
Face introduced a separate "Inference Providers" architecture in
January 2025), but the generator gave no indication of that and
prompted the user to file a bug report for an HTTP-level "not found"
response. Fully supporting HF's new Inference Providers routing
(provider selection, new endpoints/auth) would be a larger redesign of
this generator and is out of scope for this change.
Approach: add a dedicated ModelNotFoundError (a GarakException
subclass, matching this file's existing local exception pattern for
HFRateLimitException/HFLoadingException/HFInternalServerError). Raise
it when InferenceAPI._call_model receives an HTTP 404, naming the
model and pointing to Hugging Face's Inference Providers docs, instead
of falling through to the generic "unsure how to parse" TypeError. The
new exception is intentionally excluded from the @backoff retry list,
since a missing/unavailable model is not a transient condition.
Validation: ran `python -m pytest tests/generators/test_huggingface.py
-v` - all 8 tests passed, including a new test_inference_model_not_found
that mocks a 404 response with body b"Not Found" (matching the exact
response shape from the issue's traceback) and asserts
ModelNotFoundError is raised. Confirmed this test fails without the
fix (AttributeError: module has no attribute ModelNotFoundError) and
passes with it. Also ran `black --check` and a targeted pylint pass
(undefined-variable/unused-variable/unused-import) on both changed
files - both clean.
Report: NVIDIA#1297
Signed-off-by: Pujitha Paladugu <10557236+pujitha24@users.noreply.github.com>
Assisted-by: claude-sonnet-5 (via Claude Code)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Tell us what this change does. If you're fixing a bug, please mention
the github issue number.
This improves the error message raised by
huggingface.InferenceAPIwhen theHugging Face Inference API returns an HTTP 404 for a model. Previously this
surfaced as a generic
TypeError: Unsure how to parse 🤗 API response type: b'Not Found', please open an issue..., which told users to file a bug reportfor what is actually an HTTP-level "model not found" response. This is
exactly what happened in issue #1297: a maintainer identified that the model
in question is not exposed via the legacy default Inference API provider (HF
introduced a separate "Inference Providers" architecture in Jan 2025), but
the generator gave no indication of that — it just told the user to file a
bug.
Fully supporting HF's new Inference Providers routing (provider selection,
new endpoints/auth) would be a larger redesign of this generator and is out
of scope here. This change instead makes the immediate failure mode clear and
actionable: a dedicated
ModelNotFoundErroris now raised on HTTP 404,naming the model and pointing at HF's Inference Providers docs, instead of
the misleading generic parse-error message.
Please ensure you are submitting from a unique branch in your repository to
mainupstream.Verification
List the steps needed to make sure this thing works
garak -t <target_type> -n <model_name>— not run against a live Hugging Face endpoint (would require a realHF_INFERENCE_TOKENand a model that 404s under the legacy API); the new code path is covered by a targeted unit test instead (see below).python -m pytest tests/: ranpython -m pytest tests/generators/test_huggingface.py -v(all 8 tests pass, including the newtest_inference_model_not_found). Confirmed the new test fails without the fix (AttributeError: module 'garak.generators.huggingface' has no attribute 'ModelNotFoundError') and passes with it, i.e. it reproduces the reported failure mode and proves the fix. Also ranblack --checkand a targetedpylint(undefined-variable/unused-variable/unused-import) pass on the two changed files — both clean.test_inference_model_not_foundmocks a 404 response with bodyb"Not Found"(matching the exact response shape from the issue's traceback) and assertsModelNotFoundErroris raised with a message identifying the model.ModelNotFoundErroris not in the@backoff.on_exceptionretry list, so a 404 is not needlessly retried (a missing/unavailable model is not a transient condition, unlike the existing 503/rate-limit handling it sits alongside).If you are opening a PR for a new plugin that targets a specific piece of hardware or requires a complex or hard-to-find testing environment, we recommend that you send us as much detail as possible.
Specific Hardware Examples:
cuda/mps( Please notcudaviaROCmif related )Complex Software Examples:
Report: #1297
AI assistance: this change was drafted with Claude Code.
Fixes #1297