Skip to content

fix(generators): raise clear error on HF InferenceAPI 404 response - #2112

Open
pujitha24 wants to merge 1 commit into
NVIDIA:mainfrom
pujitha24:auto/issue-1297
Open

fix(generators): raise clear error on HF InferenceAPI 404 response#2112
pujitha24 wants to merge 1 commit into
NVIDIA:mainfrom
pujitha24:auto/issue-1297

Conversation

@pujitha24

Copy link
Copy Markdown

Tell us what this change does. If you're fixing a bug, please mention
the github issue number.

This improves the error message raised by huggingface.InferenceAPI when the
Hugging Face Inference API returns an HTTP 404 for a model. Previously this
surfaced as a generic TypeError: Unsure how to parse 🤗 API response type: b'Not Found', please open an issue..., which told users to file a bug report
for what is actually an HTTP-level "model not found" response. This is
exactly what happened in issue #1297: a maintainer identified that the model
in question is not exposed via the legacy default Inference API provider (HF
introduced a separate "Inference Providers" architecture in Jan 2025), but
the generator gave no indication of that — it just told the user to file a
bug.

Fully supporting HF's new Inference Providers routing (provider selection,
new endpoints/auth) would be a larger redesign of this generator and is out
of scope here. This change instead makes the immediate failure mode clear and
actionable: a dedicated ModelNotFoundError is now raised on HTTP 404,
naming the model and pointing at HF's Inference Providers docs, instead of
the misleading generic parse-error message.

Please ensure you are submitting from a unique branch in your repository to main upstream.

Verification

List the steps needed to make sure this thing works

  • Supporting configuration such as generator configuration file — not needed; change is exercised via mocked HTTP responses in unit tests, no live generator config required.
  • garak -t <target_type> -n <model_name> — not run against a live Hugging Face endpoint (would require a real HF_INFERENCE_TOKEN and a model that 404s under the legacy API); the new code path is covered by a targeted unit test instead (see below).
  • Run the tests and ensure they pass python -m pytest tests/: ran python -m pytest tests/generators/test_huggingface.py -v (all 8 tests pass, including the new test_inference_model_not_found). Confirmed the new test fails without the fix (AttributeError: module 'garak.generators.huggingface' has no attribute 'ModelNotFoundError') and passes with it, i.e. it reproduces the reported failure mode and proves the fix. Also ran black --check and a targeted pylint (undefined-variable/unused-variable/unused-import) pass on the two changed files — both clean.
  • Verify the thing does what it should — new test test_inference_model_not_found mocks a 404 response with body b"Not Found" (matching the exact response shape from the issue's traceback) and asserts ModelNotFoundError is raised with a message identifying the model.
  • Verify the thing does not do what it should not — confirmed ModelNotFoundError is not in the @backoff.on_exception retry list, so a 404 is not needlessly retried (a missing/unavailable model is not a transient condition, unlike the existing 503/rate-limit handling it sits alongside).
  • Document the thing and how it works — the raised error message itself documents the likely cause and links to https://huggingface.co/docs/inference-providers for further reading; no separate doc page seemed warranted for a single, self-explanatory error message.

If you are opening a PR for a new plugin that targets a specific piece of hardware or requires a complex or hard-to-find testing environment, we recommend that you send us as much detail as possible.

Specific Hardware Examples:

  • GPU related
    • Specific support required cuda / mps ( Please not cuda via ROCm if related )
    • Minium GPU Memory

Complex Software Examples:

  • Expensive proprietary software
  • Software with an extensive installation process
  • Software without an English language UI

Report: #1297


AI assistance: this change was drafted with Claude Code.

Fixes #1297

Motivation: huggingface.InferenceAPI raised a confusing generic
TypeError ("Unsure how to parse API response type... please open an
issue") when the Hugging Face Inference API returned an HTTP 404 for a
model. This is exactly what happened in the reported issue: the model
is not exposed via the legacy default Inference API provider (Hugging
Face introduced a separate "Inference Providers" architecture in
January 2025), but the generator gave no indication of that and
prompted the user to file a bug report for an HTTP-level "not found"
response. Fully supporting HF's new Inference Providers routing
(provider selection, new endpoints/auth) would be a larger redesign of
this generator and is out of scope for this change.

Approach: add a dedicated ModelNotFoundError (a GarakException
subclass, matching this file's existing local exception pattern for
HFRateLimitException/HFLoadingException/HFInternalServerError). Raise
it when InferenceAPI._call_model receives an HTTP 404, naming the
model and pointing to Hugging Face's Inference Providers docs, instead
of falling through to the generic "unsure how to parse" TypeError. The
new exception is intentionally excluded from the @backoff retry list,
since a missing/unavailable model is not a transient condition.

Validation: ran `python -m pytest tests/generators/test_huggingface.py
-v` - all 8 tests passed, including a new test_inference_model_not_found
that mocks a 404 response with body b"Not Found" (matching the exact
response shape from the issue's traceback) and asserts
ModelNotFoundError is raised. Confirmed this test fails without the
fix (AttributeError: module has no attribute ModelNotFoundError) and
passes with it. Also ran `black --check` and a targeted pylint pass
(undefined-variable/unused-variable/unused-import) on both changed
files - both clean.

Report: NVIDIA#1297
Signed-off-by: Pujitha Paladugu <10557236+pujitha24@users.noreply.github.com>
Assisted-by: claude-sonnet-5 (via Claude Code)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

HuggingFace InferenceAPI calls seem to no longer work

1 participant