language model pretraining

RegMix: Data Mixture as Regression for Language Model Pre-training ICLR 2025 Rethinking the Role of Text Complexity in Language Model Pretraining EMNLP 2025 SabiYarn: Advancing Low Resource Languages with Multitask NLP Pretraining ACL 2025 Sinhala Encoder-only Language Models and Evaluation ACL 2025 SOAP: Improving and Stabilizing Shampoo using Adam for Language Modeling ICLR 2025 The Learning Dynamics of Subword Segmentation for Morphologically Diverse Languages IJCNLP 2025 Untie the Knots: An Efficient Data Augmentation Strategy for Long-Context Pre-Training in Language Models ACL 2025 YuLan-Mini: Pushing the Limits of Open Data-efficient Language Model ACL 2025 A Family of Pretrained Transformer Language Models for Russian COLING 2024 Analysing The Impact of Sequence Composition on Language Model Pre-Training ACL 2024 BeanCounter: A low-toxicity, large-scale, and open dataset of business-oriented text NeurIPS 2024 Cleaner Pretraining Corpus Curation with Neural Web Scraping ACL 2024 DeepSpeed Data Efficiency: Improving Deep Learning Model Quality and Training Efficiency via Efficient Data Sampling and Routing AAAI 2024 Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Research ACL 2024 Enhancing Distractor Generation for Multiple-Choice Questions with Retrieval Augmented Pretraining and Knowledge Graph Integration ACL 2024 Fast-ELECTRA for Efficient Pre-training ICLR 2024 Generative Pretrained Structured Transformers: Unsupervised Syntactic Language Models at Scale ACL 2024 Ignore Me But Don’t Replace Me: Utilizing Non-Linguistic Elements for Pretraining on the Cybersecurity Domain NAACL 2024 Investigating Web Corpus Filtering Methods for Language Model Development in Japanese NAACL 2024 Masked Structural Growth for 2x Faster Language Model Pre-training ICLR 2024 NuNER: Entity Recognition Encoder Pre-training via LLM-Annotated Data EMNLP 2024 Oasis: Data Curation and Assessment System for Pretraining of Large Language Models IJCAI 2024 PreAlign: Boosting Cross-Lingual Transfer by Early Establishment of Multilingual Alignment EMNLP 2024 Pretraining Language Models Using Translationese EMNLP 2024 ReactXT: Understanding Molecular “Reaction-ship” via Reaction-Contextualized Molecule-Text Pretraining ACL 2024