Issues with ground truth data

[adbar]

Hi, I spotted the following issues while trying to evaluate extractors on the benchmark:

1. Annotation artifacts in the source HTML: Ground-truth pages contain wrapper tags (<marked-text>, <marked-tail>) that bias scoring against any tool that renders inline formatting.
2. Truncated ground truth: Some reference texts are partial excerpts and a more complete extraction scores lower even when it is more correct. A tool that extracts the full article scores lower than one that stops after the first section.
3. Layout tables penalized: Articles whose body sits inside <td> elements are evaluated against a reference that ignores cell boundaries, penalizing tools that handle table structure correctly, e.g. a tool that emits | paragraph | rows scores worse than one that strips the table structure.
4. Links and images not evaluated, and math absent from references. These items are invisible to the metric. A page with \(E=mc^2\) in the source has E=mc2 in the reference (rendered plain text); any tool that outputs $E=mc^2$ or $$E=mc^2$$ gets edit-distance penalized for the extra characters.

A more general comment on scoring: Edit distance favours brevity, a short extraction that closely matches a partial reference outscores a complete extraction that diverges from it.

In any case: Thank you for your contribution, ground truth data is essential in this field!

[igorsaevets]

I ran into this dataset while checking a benchmark of my own, and wanted to see which of these points can be settled by counting, so I measured the published WebMainBench_545.jsonl directly. Counts over all 545 samples; script linked at the end.

Links and images. The references contain almost none.

in groundtruth_content	samples	share
Markdown hyperlink [t](u)	3	0.6%
Markdown image 	4	0.7%
bare URL as plain text	102	18.7%
heading	443	81.3%

Destinations do survive, as unlinked text. Under edit distance that penalises an extractor for exactly the characters that make a link a link.

Layout tables. Using the dataset's own meta.table labels as the denominator: of the 28 pages labelled as containing a layout table, 1 reference contains any | row. Of the 179 labelled as containing any table, 39 do.

Maths, which does not hold up. Of the 257 pages labelled as containing an equation, 230 references (89.5%) do contain a maths delimiter. Whatever is happening there looks per-sample rather than systematic, so I do not think the general form of that point survives.

One thing not in the list above. In the published 545 file, main_html is empty in 445 rows (81.7%) and convert_main_content in 200 (36.7%), although the README documents both as available for all 7,809 samples. For most of the calibrated subset it is therefore not possible to compare a reference against the annotated subtree it came from, which is the check that would settle the truncation question either way.

Script, and the two mistakes I made and fixed while writing it (meta.table is an array of kinds rather than a boolean, so a first pass reading it as a boolean scores every category zero; and a naive link detector counts ![](...) image embeds as hyperlinks):

https://github.com/igorsaevets/page2ai-benchmark/blob/master/webmainbench-groundtruth-audit.mjs

Happy to be told I am miscounting something. Disclosure: I maintain a small HTML-to-Markdown library, which is how I ended up here.

[igorsaevets]

Correcting my own comment above. An adversarial review of my script found three problems, and I re-measured each before accepting it. Updated script and output: igorsaevets/page2ai-benchmark@a976729

1. The main_html point was wrong, and it is the one that mattered. I wrote that for most of the subset it is not possible to compare a reference against the annotated subtree. It is possible. All 545 rows carry the full annotated html, 444 of the 445 blank rows still carry the cc-select attribute, and this repository already ships webmainbench/utils/main_html.py with extract_main_html() to rebuild the subtree from exactly that. So the blank fields are a missing derived output and an undocumented reconstruction step, not missing evidence. That does mean the truncation question in the original issue is answerable from the published file, which is a better outcome than what I said.

2. The bare-URL figure of 102 was too high. My detector was a plain https?://\S+ over the whole document, which also matches URLs inside fenced code, inline code, raw HTML attributes and Markdown link targets. Excluding those contexts gives 81. And "destinations survive as unlinked text" was more than my detector established: it measures syntax presence, not whether a specific destination in the source made it through.

3. The table figure of 39 included one pipe row inside a fenced block. Outside fences it is 38 of the 179 table-labelled samples. The layout-table figure is unchanged at 1 of 28.

Unchanged: link syntax in 3 of 545, image syntax in 4 of 545, maths delimiters in 230 of 257 equation-labelled samples.

One thing I overstated in framing rather than in arithmetic: the maths result refutes only the broad reading that maths is generally absent from the references. It does not measure per-formula recall or fidelity, so it does not speak to the E=mc² example specifically. These are all presence checks on the reference text, not recall measurements.

[khowardskennel-rgb]

in**