Skip to content

Words lose their separating spaces in 7.21.0 ("Related Work" -> "RelatedWork"), bisected to #351 #355

Description

@YHC66

What happens

Since 7.21.0, some word boundaries that 7.20.0 recovered are lost, most visibly in bold section headings of LaTeX papers. Using 2305.03393v1-pg9.pdf from docling's test data (file):

import sys
from docling_core.types.doc.page import TextCellUnit
from docling_parse.pdf_parser import DoclingPdfParser

doc = DoclingPdfParser(loglevel="fatal").load(path_or_stream=sys.argv[1])
for page_no, page in doc.iterate_pages():
    for c in page.iterate_cells(unit_type=TextCellUnit.LINE):
        if "Optimization" in c.text:
            print(repr(c.text))
docling-parse output
7.20.0 'Hyper Parameter Optimization'
7.21.0 '5.1 HyperParameterOptimization'

The same thing shows up across docling's test PDFs (line cells, 7.20.0 -> 7.21.0):

  • 2305.03393v1.pdf: Related Work -> 2 RelatedWork, Problem Statement -> 3 ProblemStatement, Fig. 1. -> Fig.1. Comparison between ..., Fig. 3. -> Fig.3. ..., Language Definition -> LanguageDefinition
  • 2305.03393v1-pg9.pdf: Quantitative Results -> QuantitativeResults
  • 2203.01017v2.pdf: Implementation Details -> ImplementationDetails, custom OCR -> customOCR, problem where -> problemwhere
  • redp5110_sampled.pdf: dotted TOC leaders get attached to the preceding word (Roles., duties.)

In docling this makes tests/test_interfaces.py::test_convert_path, ::test_convert_stream and tests/test_heading_hierarchy_pdf.py::test_pdf_pipeline_assigns_heading_levels_from_existing_fixture fail against their ground truth. docling's CI doesn't see it because its uv.lock still pins 7.20.0, but pip install docling resolves docling-parse>=7.20.0,<8.0.0 to 7.21.0.

Bisect

Built from source on macOS arm64 (Python 3.12):

So the change comes from the cell-contraction rewrite in #351. Most of the affected lines are short, and the lost gaps are the ones between words of a single bold run. That fits the new line-local threshold (two-cluster k-means over normalized gaps) putting real word gaps in the "intra-word" cluster when a line has few gaps and no explicit space glyphs. That's a guess from reading the PR description, though; I haven't traced it in the code.

Environment

  • docling-parse 7.21.0 (PyPI wheel) and a source build of e64d838
  • macOS 26 arm64, Python 3.12

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions