pretraining data

On The Origin of Cultural Biases in Language Models: From Pre-training Data to Linguistic Phenomena NAACL 2025 Optimizing Pre-Training Data Mixtures with Mixtures of Data Expert Models ACL 2025 Overestimation in LLM Evaluation: A Controlled Large-Scale Study on Data Contamination’s Impact on Machine Translation ICML 2025 Procedural Knowledge in Pretraining Drives Reasoning in Large Language Models ICLR 2025 SampleMix: A Sample-wise Pre-training Data Mixing Strategy by Coordinating Data Quality and Diversity EMNLP 2025 TokenSmith: Streamlining Data Editing, Search, and Inspection for Large-Scale Language Model Training and Interpretability EMNLP 2025 Training Bilingual LMs with Data Constraints in the Targeted Language ACL 2025 Training on the Benchmark Is Not All You Need AAAI 2025 UnitCoder: Scalable Code Synthesis from Pre-training Corpora EMNLP 2025 AboutMe: Using Self-Descriptions in Webpages to Document the Effects of English Pretraining Data Filters ACL 2024 “According to . . . ”: Prompting Language Models Improves Quoting from Pre-Training Data EACL 2024 A Pretrainer’s Guide to Training Data: Measuring the Effects of Data Age, Domain Coverage, Quality, & Toxicity NAACL 2024 Code Needs Comments: Enhancing Code LLMs with Comment Augmentation ACL 2024 Cross-Care: Assessing the Healthcare Implications of Pre-training Data on Language Model Bias NeurIPS 2024 Data Filtering Networks ICLR 2024 Data Mixture Inference Attack: BPE Tokenizers Reveal Training Data Compositions NeurIPS 2024 Deciphering the Impact of Pretraining Data on Large Language Models through Machine Unlearning ACL 2024 DecorateLM: Data Engineering through Corpus Rating, Tagging, and Editing with Language Models EMNLP 2024 Detection and Measurement of Syntactic Templates in Generated Text EMNLP 2024 DPDLLM: A Black-box Framework for Detecting Pre-training Data from Large Language Models ACL 2024 Dual Operating Modes of In-Context Learning ICML 2024 Formality is Favored: Unraveling the Learning Preferences of Large Language Models on Data with Conflicting Knowledge EMNLP 2024 IndicLLMSuite: A Blueprint for Creating Pre-training and Fine-Tuning Datasets for Indian Languages ACL 2024 Investigating Data Contamination in Modern Benchmarks for Large Language Models NAACL 2024 KidLM: Advancing Language Models for Children – Early Insights and Future Directions EMNLP 2024