What Does a Language Model Know If It Has Never Seen the Future? Temporal Isolation as a Natural Experiment in Knowledge Attribution
Abstract
AbstractHow much of a language model’s apparent knowledge reflects general linguistic competence, and how much reflects memorisation of facts seen during pretraining? Disentangling the two is normally confounded by data contamination, because modern corpora overlap with the benchmarks used to evaluate them. We exploit a natural experiment that removes exact benchmark exposure by construction. TALKIE-1930 is a 13B-parameter transformer trained exclusively on approximately 260 billion tokens of English published before 1931; TALKIE-WEB is an architecture-, tokeniser-, and compute-matched twin trained on an equivalent volume of FineWeb. Because every modern NLP benchmark postdates 1930, any post-1930 capability TALKIE-1930 exhibits cannot arise from benchmark contamination. We study this model pair along three axes. First, syntactic competence transfers across eras with only a modest gap (BLiMP: 76.9% for TALKIE-1930 vs. 82.0% for TALKIE-WEB), and an OCR-noise ablation shows that this gap is not an artefact of character-level corruption in the historical corpus. Second, in-context learning is largely preserved by temporal isolation: TALKIE-1930 matches TALKIE-WEB on SST-2 (91.2% vs. 90.9% at k=32), exceeds it on MNLI (47.3% vs. 37.8%), and learns post-2006 genres it cannot have seen (tweet sentiment and emotion) with learning-rate slopes statistically indistinguishable from TALKIE-WEB’s. Third, linear probes recover the temporal provenance of a factual statement from hidden states at 98.2% accuracy (TALKIE-1930) and 99.5% (TALKIE-WEB); a three-way probe on TALKIE-1930 that adds a suspected-leaked class reaches 96.0% accuracy (chance approximately 33%), well above lexical and permutation baselines, though this figure is carried mainly by the temporal contrast and the leaked class is only partially recovered. Finally, we document a dissociation between representation and behaviour: although TALKIE-1930’s hidden states linearly encode temporal-boundary information, its gr