Skip to content
conftrace_
2026 EMNLP EMNLP 2026

Data Shapes What, Scale Shapes How: Lexical Learning Dynamics in LLMs

AbstractWe investigate how model scale shapes lexical acquisition in LLMs, introducing trajectory comparison as a methodological tool and three new convergence metrics. Tracking surprisal dynamics across four Pythia model sizes and 153 training checkpoints, we report two main findings. First, scale drives a global rise in inter-word surprisal coherence: synonym curves become more correlated at larger scales, but baseline correction reveals that the advantage of semantic relatedness over random pairs is invariant with scale, indicating that what scales is broad inter-word coherence rather than semantic abstraction specifically. Second, learning dynamics are shaped primarily by training data rather than by model capacity: surprisal trajectories are highly similar across model sizes, and our new metrics confirm that larger models converge faster but not along qualitatively different paths. Together, these results call for baseline-corrected analyses and longitudinal evaluation when assessing lexical competence in LLMs.

More from EMNLP 2026

All EMNLP 2026 papers →