How Many Duplicates Are Left? Capture-Recapture and Sampling Inversion for Residual Duplication in FineWeb
Abstract
AbstractPretraining corpus releases report how much duplication their pipelines removed, but we are not aware of one that estimates how much they missed. Measuring the residue is difficult because two recall problems apply at once: detectors miss duplicates they cannot see, and any tractable subsample misses duplicates whose partners were not sampled. In this paper we address both. We 1) run four detectors with distinct failure modes over a corpus and treat them as capture occasions, 2) recover what all four missed through capture-recapture, and 3) invert the sampling rate that produced the subsample, so that the result describes a full 15-trillion-token corpus rather than a convenient slice of it. The design is scored first against a planted population of known size, where it recovers 89.2% of the truth from a 44.5% detected rate while remaining a lower bound. Applied to the 14,868,862-document sample-10BT release of FineWeb, 4.50% of probes have a surviving near-duplicate, and capture-recapture raises that to 4.81% (95% CI 4.37-5.26). Inverting a sampling rate of one document in 1,532 then carries the figure to the corpus and gives a lower bound of 68 on the mean near-duplicate degree of a FineWeb document. Near-duplicate here means lexical overlap at Jaccard 0.35 or above. The bound is on a mean and not on a prevalence, since our design leaves the share of documents with any surviving duplicate anywhere between 7% and 73%. The survivors also carry a positional signature. The two copies of an exact-duplicate pair sit a mean 34.4% of the corpus apart against 33.3% expected under uniform sampling, which supports reading them as residue of FineWeb’s deliberate per-snapshot deduplication. A deduplicated corpus, therefore, is not a duplicate-free corpus, and the size of the gap can now be measured in one streaming pass.