Pretraining

AfriqueLLM: How Data Mixing and Model Architecture Impact Continued Pre-training for African Languages ACL 2026 Corpus-Dependent Subcharacter Encoding via HMM-Guided Code Assignment ACL 2026 Enhancing Linguistic Competence of Language Models through Pre-training with Language Learning Tasks ACL 2026 From Where Words Come: Efficient Regularization of Code Tokenizers Through Source Attribution ACL 2026 How Training Data Shapes the Use of Parametric and In-Context Knowledge in Language Models ACL 2026 SyntaxMind at SemEval-2026 Task 6: Exploring Transformers and LLMs for Unmasking Political Question Evasions SemEval 2026 Cantonese Punctuation Restoration using LLM Annotated Data Interspeech 2025 Continued Pretraining and Interpretability-Based Evaluation for Low-Resource Languages: A Galician Case Study ACL 2025 Data-Efficient Selection via Grammatical Complexity in Continual Pre-training of Domain-Specific LLMs EMNLP 2025 DELTA: Pre-Train a Discriminative Encoder for Legal Case Retrieval via Structural Word Alignment AAAI 2025 Designing and Contextualising Probes for African Languages ACL 2025 Diversity-Rewarded CFG Distillation ICLR 2025 Dynamic Context-Aware Streaming Pretrained Language Model For Inverse Text Normalization Interspeech 2025 Instruction-Tuning Data Synthesis from Scratch via Web Reconstruction ACL 2025 LP Data Pipeline: Lightweight, Purpose-driven Data Pipeline for Large Language Models EMNLP 2025 On the Path to Make Ukrainian a High-Resource Language ACL 2025 Training compute-optimal transformer encoder models EMNLP 2025 Unveiling the Potential of BERT-family: A New Recipe for Building Scalable, General and Competitive Large Language Models ACL 2025 VisualWebInstruct: Scaling up Multimodal Instruction Data through Web Search EMNLP 2025 DFormer: Rethinking RGBD Representation Learning for Semantic Segmentation ICLR 2024 MathPile: A Billion-Token-Scale Pretraining Corpus for Math NeurIPS 2024 OtoBERT: Identifying Suffixed Verbal Forms in Modern Hebrew Literature EMNLP 2024 REFeREE: A REference-FREE Model-Based Metric for Text Simplification COLING 2024 Representation Deficiency in Masked Language Modeling ICLR 2024 A Survey on Model Compression and Acceleration for Pretrained Language Models AAAI 2023