fix(hf): set whitespace_flexible=True in grammar_from_json_schema to prevent silent array collapse

[planetf1]

Summary

Closes #1510 — fix silent array truncation in LocalHFBackend structured output.

Why

LocalHFBackend was calling llguidance.LLMatcher.grammar_from_json_schema with defaults={"whitespace_flexible": False} at three call sites. This compiles a compact-JSON grammar (no whitespace between tokens). Under greedy decoding, this puts the model in a state where the highest-probability grammar-compatible token immediately after opening an array [ closes it early — either immediately (], producing {"result":[]}) or after the first item.

The result is a silent wrong answer: no exception is raised, the output is schema-valid, and the caller cannot detect the failure.

The False was introduced in PR #288 as a mechanical outlines → llguidance port and was never intentional. llguidance's own _lib.pyi stub documents that whitespace_flexible defaults to True when defaults is omitted — verified empirically (byte-for-byte identical compiled grammar to passing True explicitly). So the bug was scoped to exactly the three call sites that explicitly passed False; a fourth call site, in chat_completion_request_to_transformers_inputs (the m serve OpenAI-compatible path), already omitted defaults entirely and was never affected by #1510.

What changed

File	Change
mellea/backends/huggingface.py	whitespace_flexible: False → True at all three grammar_from_json_schema call sites (_generate_from_context_with_kv_cache, _generate_from_context_standard, _generate_from_raw); explanatory comment referencing #1510; imports the shared _LLGUIDANCE_GRAMMAR_DEFAULTS constant from util.py instead of redefining it
mellea/formatters/granite/base/util.py	Defines _LLGUIDANCE_GRAMMAR_DEFAULTS (single source of truth). The fourth call site, in chat_completion_request_to_transformers_inputs, was not affected by #1510 (it already got llguidance's implicit True default) — hardened anyway with an explicit overrides= pass, since this endpoint takes its schema straight off the wire and a caller could otherwise embed their own x-guidance.whitespace_flexible: false to defeat a defaults= pass. The three huggingface.py sites use defaults= since their schemas come from local Pydantic models, not caller-controlled input
test/backends/test_huggingface_unit.py	Four regression tests — one per call site, mock-based, no real model required; assert whitespace_flexible=True is passed. The fourth test exercises util.py's function but lives here deliberately, alongside the other three #1510 regression tests, rather than in a new dedicated test file
docs/docs/integrations/huggingface.md	New "Constrained decoding" section documenting the whitespace_flexible=True behaviour and the measured token overhead

Before / After

# Before — silent truncation, no exception:
result = await session.aact(format=SectionTitles, ...)
# → SectionTitles(sections=[])                         # 3b: empty array
# → SectionTitles(sections=["Introduction"])           # 8b: partial list

# After — correct output:
result = await session.aact(format=SectionTitles, ...)
# → SectionTitles(sections=["Introduction", "Background", "Methods", ...])

Empirical evidence

Tested across three models, three devices, 10 trials, 5 schema variants, n=1–8 items:

Model	Device	wf=False OK	wf=True OK
granite-4.1-3b	CPU	100/250	250/250
granite-4.1-3b	CUDA	100/250	250/250
granite-4.1-3b	MPS	100/250	250/250
granite-4.1-8b	CPU	190/250	250/250
granite-4.1-8b	CUDA	210/250	250/250
granite-4.1-8b	MPS	190/250	250/250
granite-4.0-micro	CUDA	190/250	230/250

wf=True is clean for 3b and 8b on all devices. The 20 wf=True failures on micro are single_str_field n=2 producing 1 item instead of 2 on every trial — a model-level instruction-following limitation on that schema, not caused by this bug.

Token overhead

wf=True produces spaced JSON, using ~1.5× more tokens for the same content. Callers with a tight explicit ModelOption.MAX_NEW_TOKENS budget should size accordingly. Generation that runs to EOS still completes, but consumes proportionally more tokens getting there.

Testing

uv run pytest test/backends/test_huggingface_unit.py -x -q
# 60 passed, 3 warnings in 7.73s

uv run mypy mellea/backends/huggingface.py mellea/formatters/granite/base/util.py test/backends/test_huggingface_unit.py
# no issues

uv run ruff format mellea/backends/huggingface.py mellea/formatters/granite/base/util.py test/backends/test_huggingface_unit.py
uv run ruff check mellea/backends/huggingface.py mellea/formatters/granite/base/util.py test/backends/test_huggingface_unit.py
# all clean

uv run pytest test/ -m "not qualitative" -q
# 3860 passed, 21 skipped, 134 deselected, 2 xfailed, 2 xpassed in 641.60s

The four new regression tests each fail on their respective pre-fix code and pass on the fix.

[planetf1]

@nrfulton @jakelorocco any thoughts on this? See issue for more details - it came up when debugging some unexpected responses with melee

[jakelorocco]

This seems like a reasonable fix.

However, we did previously hit a lot of issues with json generation where the model would generate a large amount of whitespace and then run out of tokens.

@planetf1, would you be able to run your test set of 250 tests with a few other parameters to see if other options are able to keep performance?

It looks like llguidance has a far number of parameters that we could potentially tweak to get the performance improvements you point out without risking unlimited whitespace: https://github.com/guidance-ai/llguidance/blob/main/docs/json_schema.md#whitespace-handling.

I think changing the default key / item separators might work. whitespace_pattern seems like a very promising option though where we could basically mimic the whitespace_flexible pattern but with an upper bound on the number of whitespace tokens allowed.

From the docs:

whitespace_flexible: true is equivalent to whitespace_pattern: r"[\x20\x0A\x0D\x09]+"

And we could do:

r"[\x20\x0A\x0D\x09]{1,8}"

I'll approve; but if you don't have time to run additional tests, could you please open an issue about investigating this upper bounding / additional parameters and publish your qualitative tests somewhere?