Investigate bounding llguidance whitespace_pattern for LocalHFBackend structured output

[planetf1]

Summary

Follow-up to #1510 / PR #1513, from Jake Lorocco's review comment: investigate bounding the amount of whitespace LocalHFBackend can generate under whitespace_flexible=True, and publish the raw empirical dataset behind PR #1513's fix.

Background

PR #1513 sets whitespace_flexible=True on all llguidance.LLMatcher.grammar_from_json_schema call sites to fix #1510 (silent array truncation). This permits unbounded whitespace between JSON tokens, which is safe from a correctness standpoint but costs ~1.5x more output tokens than compact JSON for the same content.

Jake raised a prior incident where json generation produced large amounts of whitespace and ran out of tokens, and suggested bounding it via llguidance's whitespace_pattern option instead of the unbounded whitespace_flexible=True:

whitespace_flexible: true is equivalent to whitespace_pattern: r"[\x20\x0A\x0D\x09]+"

And we could do: r"[\x20\x0A\x0D\x09]{1,8}"

What needs investigating

I tested this suggestion directly against llguidance==1.7.3 (the version currently pinned in this repo) using a byte-level tokenizer/matcher and found the naive bound does not work: feeding 200 consecutive space tokens after an open JSON array ([) resulted in all 200 being accepted, across three configurations:

1. whitespace_flexible=True + whitespace_pattern=r"[\x20\x0A\x0D\x09]{1,8}"
2. whitespace_pattern alone (no whitespace_flexible key at all)
3. whitespace_flexible=False + whitespace_pattern=r"[\x20\x0A\x0D\x09]{1,8}"

In none of these did the {1,8} upper bound reject the 9th, 50th, or 200th consecutive space token. Either the bound is being compiled at a grammar scope this test didn't reach (e.g. it may only cap whitespace within a single separator, not prevent the model from repeatedly satisfying-then-reopening a whitespace-accepting state), or there's a version-specific gap between llguidance's documented behaviour and its compiled output. This needs someone with more llguidance internals context to dig into the actual compiled grammar, not just the observed acceptance mask.

Action items

• Determine why whitespace_pattern's upper bound isn't enforced in the tested configuration (or whether a different llguidance API/version is needed)
• If a working bound is found, benchmark it against the existing 250-trial correctness dataset from PR fix(hf): set whitespace_flexible=True in grammar_from_json_schema to prevent silent array collapse #1513 to confirm it doesn't reintroduce the llguidance whitespace_flexible=False can silently collapse constrained JSON arrays to empty (LocalHFBackend) #1510 collapse
• Publish the raw per-trial data behind PR fix(hf): set whitespace_flexible=True in grammar_from_json_schema to prevent silent array collapse #1513's empirical evidence table (currently only the aggregated pass/fail counts are in the PR description) — needed so this and future whitespace-tuning work can be verified against the same trials rather than re-run from scratch

[planetf1]

looking into this (lower priority) as a followup to 1513