Skip to content
conftrace_
2026 EMNLP EMNLP 2026

Extraction Changes the Route: Main-Content Extraction and Language Identification in Web Corpora

AbstractMain-content extraction commonly precedes language identification (LID) and determines its input in web-corpus pipelines. We measure this dependency on 7,808 WebMainBench pages using three extractors and three LID systems. We compare human-selected main text with annotated-region and full-page extractor outputs. A context flip occurs when annotated-region extraction preserves the route on human-selected text but full-page extraction changes it. For pages with nonempty text and matching human-selected and annotated-region routes, context-flip rates are 1.10–3.20%. On the common eligible set, Magic-HTML has lower paired rates than Trafilatura for all three LID systems, with each corresponding 95% interval excluding zero. All-page route loss counts empty outputs or routes that differ from the route on human-selected text. Magic-HTML’s empty full-page outputs add 5.84 percentage points. The fixed snapshot’s coarse non_en–en difference is positive in all nine configurations and remains positive after standardization to the page-complexity field (meta.level). A 100-page audit finds 35 unsupported, 29 supported, and 36 indeterminate routes. Of 35 unsupported flips, 26 move from predicted English to another route, and 17 involve changed content selection alone or in mixed cases. Among supported flips, 21 move to English. Extraction context can change text availability and downstream routing.

More from EMNLP 2026

All EMNLP 2026 papers →