conftrace_

document concatenation

2 papers

Explore in graph

Co-occurring keywords · papers mentioning both

causal masking (1) knowledge memorization (1) sequence composition (1) hyper-link referral (1) web page packing (1) context utilization (1) language model pretraining (1) long-context training (1) data pipeline (1) pretraining corpus (1)

Papers

Scaling Long Context Training Data by Long-Distance Referrals ICLR 2025 Analysing The Impact of Sequence Composition on Language Model Pre-Training ACL 2024