fix(chunker): respect contextualized table token limits

[pablopupo]

• Apply the HybridChunker metadata-adjusted budget when splitting Markdown table rows.
• Count each fully contextualized chunk and reduce the table budget by its measured overflow.
• Preserve lossless, terminating line splits when token counts change after prefix concatenation.
• Add regressions for contextualized overflow, non-additive token boundaries, Unicode prefixes, and table reconstruction.

Resolves docling-project/docling#3428

Verification

• uv run --no-sync pytest --cov=docling_core --cov-report=xml test: 604 passed, 6 skipped
• pre-commit run --all-files
• .github/scripts/build-packages.sh
• Supplied PDF: maximum contextualized chunk 8,189 / 8,192; both tables reconstructed byte-for-byte

[github-actions]

✅ DCO Check Passed

Thanks @pablopupo, all your commits are properly signed off. 🎉

[mergify]

Merge Protections

🔴 1 of 2 protections blocking · waiting on 👀 reviews

	Protection	Waiting on
🔴	Require two reviewer for test updates	👀 reviews
🟢	Enforce conventional commit	—

🔴 Require two reviewer for test updates

Waiting for

• #approved-reviews-by >= 2

This rule is failing.

When test data is updated, we require two reviewers

• #approved-reviews-by >= 2

---

Show 1 satisfied protection

🟢 Enforce conventional commit

Make sure that we follow https://www.conventionalcommits.org/en/v1.0.0/

• title ~= ^(fix|feat|docs|style|refactor|perf|test|build|ci|chore|revert)(?:\(.+\))?(!)?:

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!