test(agent-prompt): gate the documented prompt budget against the enforced one (#585) - #586
Draft
padak wants to merge 1 commit into
Draft
test(agent-prompt): gate the documented prompt budget against the enforced one (#585)#586padak wants to merge 1 commit into
padak wants to merge 1 commit into
Conversation
…orced one (#585) The 62 000 B ceiling on keboola-expert.md was already enforced by test_agent_prompt_under_token_budget, but nothing kept the prose in sync with it. CONTRIBUTING.md (3 sites) and kbagent-pr-reviewer.md still claimed a 60 KB budget -- stale since v0.48.0 raised the cap -- so an author trimming to the documented figure trimmed ~2 kB more than needed, and a reviewer checking against it would pass a file CI would reject. Add test_documented_budget_matches_enforced_budget, which asserts every doc site quotes PROMPT_BYTE_BUDGET and carries no stale 60 KB claim, and correct the four sites. Also spell the literal as plain digits in a comment: the PEP 515 form 62_000 is invisible to 'grep -rn 62000', which is why the budget was reported as unenforced in the first place.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds
test_documented_budget_matches_enforced_budgetand corrects the four doc sites that quoted a stale prompt-budget figure.Why — a correction to the issue's premise first
The issue reports that the 62 000 B budget on
plugins/kbagent/agents/keboola-expert.mdis unenforced. It is enforced, and has been since v0.48.0:That test is in the per-PR CI suite (
ci.yml,pull_requesttrigger), andactions/checkoutchecks out the merge commit — so it already fails the merge, which is the scenario the issue is about. The proposed new test would have duplicated it.The reason
grep -rn 62000 scripts/ tests/ .github/ Makefilefound nothing: the constant is spelledPROMPT_BYTE_BUDGET = 62_000with a PEP 515 underscore separator, which that pattern cannot match.Why — the real defect underneath
Chasing that miss surfaced a genuine drift. Four places state the budget in prose, and all four were stale:
CONTRIBUTING.md:381CONTRIBUTING.md:464CONTRIBUTING.md:594plugins/kbagent/agents/kbagent-pr-reviewer.md:138v0.48.0 raised the ceiling 60 kB → 62 kB in the test and left every doc site behind. The file sits at 61 990 B today — over the documented budget, under the enforced one. So the prose was actively misleading in both directions: an author trimming to 60 KB cut ~2 kB of signal for nothing, and
/kbagent:review(which readskbagent-pr-reviewer.md) was telling reviewers the wrong number.Changes
tests/test_agent_prompt.py— newtest_documented_budget_matches_enforced_budget: every path inBUDGET_DOC_SITESmust quotePROMPT_BYTE_BUDGETas62 000 Band must not carry a60 KB prompt budgetclaim. Raising the cap now forces the docs to move with it.tests/test_agent_prompt.py— comment spells the literal in plain digits sogrep 62000hits, and names the constant as the single source of truth.CONTRIBUTING.md— three sites corrected; the first now also names the enforcing test, notes that CI builds the merge commit, and giveswc -cas the pre-flight check.plugins/kbagent/agents/kbagent-pr-reviewer.md— corrected and pointed at the test.No behavior change; docs and test only. No version bump, no changelog entry.
How it was tested
make check— clean: lint, format, changelog-check, 5490 passed, 12 skipped.CONTRIBUTING.mdsite to60 KBmakes it fail with the drifted path named; restoring makes it pass. A drift gate that cannot fail is worthless, so this was verified rather than assumed.wc -c), within budget.Note on headroom
The issue flags ~15 bytes free. Real headroom on this branch is 10 bytes (61 990 of 61 999 usable — the assertion is strict
<). The next addition to that file hits the ceiling immediately; this PR does not change that, it only makes the number people trim against the correct one. Splittingkeboola-expert.mdinto per-domain specialists is the standing recommendation in the test's own comment and is out of scope here.Fixes #585