Skip to content
conftrace_
2026 EMNLP EMNLP 2026

Silent Structured-Content Corruption in Document Parsing: A Span-Typed Diagnostic and a Released Cross-Parser Resource

AbstractDocument-AI pipelines increasingly begin by converting a PDF to text with a parser—classical extractors, OCR, layout models, or vision-language models—before any retrieval or reasoning. Overall-quality metrics (edit distance, TEDS) score these parsers as if all characters mattered equally. They do not: a corrupted number, date, identifier, or table cell changes meaning while barely moving a similarity score. We isolate this failure with a span-typed diagnostic over six parsers, ten structured span types, and two conditions (digital and scanned), on a controlled corpus with exact ground truth plus a real arXiv subset. Three findings. (i) Preservation is highly parser- and type-dependent: the best parser preserves 99.4% of spans while others fail specific types catastrophically (Docling drops markup to 6.0%; MarkItDown returns nothing on scanned pages), so “which parser” is a per-span-type risk decision, not a single score. (ii) Document-level text similarity does not predict span loss (Spearman ρ ≤ 0.46 across all parsers and conditions; ≈ 0 for VLMs), so standard metrics are blind to it. (iii) Downstream, corrupted spans are dangerous precisely when the parse looks successful: among questions whose evidence span was corrupted but whose document still returned text, 96.5% are answered with a confident wrong value rather than an abstention—an undetectable error—whereas total parse failure is comparatively safe because it forces an abstention. We release the resource behind these numbers: 566,904 per-span verdicts over 660 documents and 7,560 parser outputs, under CC-BY, with an independent verifier (49,720 checks) that re-derives every reported number.

More from EMNLP 2026

All EMNLP 2026 papers →