language model pretraining

Beyond Random Sampling: Efficient Language Model Pretraining via Curriculum Learning EACL 2026 Contextual morphologically-guided tokenization for Latin encoder models EACL 2026 Demystifying Mixed Outcomes of Self-Training: Pre-training Analyses on Non-Toy LLMs EACL 2026 Improving Romanian LLM Pretraining Data using Diversity and Quality Filtering EACL 2026 KoCo: Conditioning Language Model Pre-training on Knowledge Coordinates ACL 2026 NE-BERT: A Multilingual Language Model for Nine Northeast Indian Languages EACL 2026 Qomhrá: A Bilingual Irish and English Large Language Model EACL 2026 Scaling Data-Constrained Language Models with Synthetic Data EACL 2026 AdamS: Momentum Itself Can Be A Normalizer for LLM Pretraining and Post-training EMNLP 2025 An Expanded Massive Multilingual Dataset for High-Performance Language Technologies (HPLT) ACL 2025 Beyond Fixed Length: Bucket Pre-training is All You Need IJCAI 2025 Craw4LLM: Efficient Web Crawling for LLM Pretraining ACL 2025 Crypto-LLM: Two-Stage Language Model Pre-training with Ciphered and Natural Language Data IJCNLP 2025 DEPT: Decoupled Embeddings for Pre-training Language Models ICLR 2025 Drop Dropout on Single Epoch Language Model Pretraining ACL 2025 Enhancing Elusive Clues in Knowledge Learning by Contrasting Attention of Language Models AAAI 2025 FaBERT: Pre-training BERT on Persian Blogs NAACL 2025 Improving Cross Lingual Transfer by Pretraining with Active Forgetting EMNLP 2025 Influence-driven Curriculum Learning for Pre-training on Limited Data EMNLP 2025 Learning Dynamics of Meta-Learning in Small Model Pretraining IJCNLP 2025 MIND: Math Informed syNthetic Dialogues for Pretraining LLMs ICLR 2025 Multilingual Language Model Pretraining using Machine-translated Data EMNLP 2025 On the Path to Make Ukrainian a High-Resource Language ACL 2025 Overlapping Context with Variable-Length Stride Increases Diversity when Training Large Language Model for Code ACL 2025 Perplexed by Perplexity: Perplexity-Based Data Pruning With Small Reference Models ICLR 2025