Papers
RecombiText: Compositional Data Augmentation for Enhancing LLM Pre-Training Datasets in Low-Resource Scenarios
Alexander Tampier, Lukas Thoma, Loris Schoenegger et al.
MoEP: Modular Expert Paths for Sample-Efficient Language Modeling
Joonas Tapaninaho
Masked Diffusion Language Models with Frequency-Informed Training
Despoina Kosmopoulou, Efthymios Georgiou, Vaggelis Dorovatas et al.
Pretraining Language Models with LoRA and Artificial Languages
Nalin Kumar, Mateusz Lango, Ondrej Dusek
Batch-wise Convergent Pre-training: Step-by-Step Learning Inspired by Child Language Development
Ko Yoshida, Daiki Shiono, Kai Sato et al.
Linguistic Units as Tokens: Intrinsic and Extrinsic Evaluation with BabyLM
Achille Fusco, Maria Letizia Piccini Bianchessi, Tommaso Sgrizzi et al.
Active Curriculum Language Modeling over a Hybrid Pre-training Method
Eleni Fysikoudi, Sharid Loáiciga, Asad B. Sayeed
You are an LLM teaching a smaller model everything you know: Multi-task pretraining of language models with LLM-designed study plans
Wiktor Kamzela, Mateusz Lango, Ondrej Dusek
Once Upon a Time: Interactive Learning for Storytelling with Small Language Models
Jonas Mayer Martins, Ali Hamza Bashir, Muhammad Rehan Khalid et al.
Mask and You Shall Receive: Optimizing Masked Language Modeling For Pretraining BabyLMs
Lukas Edman, Alexander Fraser
CLASS-IT: Conversational and Lecture-Aligned Small-Scale Instruction Tuning for BabyLMs
Luca Capone, Alessandro Bondielli, Alessandro Lenci
Dialogue Is Not Enough to Make a Communicative BabyLM (But Neither Is Developmentally Inspired Reinforcement Learning)
Francesca Padovani, Bastian Bunzeck, Manar Ali et al.
Findings of the Third BabyLM Challenge: Accelerating Language Modeling Research with Cognitively Plausible Data
Lucas Charpentier, Leshem Choshen, Ryan Cotterell et al.
Understanding and Enhancing Mamba-Transformer Hybrids for Memory Recall and Language Modeling
Hyunji Lee, Wenhao Yu, Hongming Zhang et al.
Influence-driven Curriculum Learning for Pre-training on Limited Data
Loris Schoenegger, Lukas Thoma, Terra Blevins et al.
Teacher Demonstrations in a BabyLM’s Zone of Proximal Development for Contingent Multi-Turn Interaction
Suchir Salhan, Hongyi Gu, Donya Rooein et al.
Single layer tiny Co4 outpaces GPT-2 and GPT-BERT
Noor Ul Zain, Mohsin Raza Naseem, Ahsan Adeel
Do Syntactic Categories Help in Developmentally Motivated Curriculum Learning for Language Models?
Arzu Burcu Güven, Anna Rogers, Rob Van Der Goot
A Morpheme-Aware Child-Inspired Language Model
Necva Bölücü, Burcu Can
CurLL: A Developmental Framework to Evaluate Continual Learning in Language Models
Pavan Kalyan Tankala, Shubhra Mishra, Satya Lokam et al.
Beyond Repetition: Text Simplification and Curriculum Learning for Data-Constrained Pretraining
Matthew Theodore Roque, Dan John Velasco
What did you say? Generating Child-Directed Speech Questions to Train LLMs
Whitney Poh, Michael Tombolini, Libby Barak
Two ways into the hall of mirrors: Language exposure and lossy memory drive cross-linguistic grammaticality illusions in language models
Kate McCurdy, Katharina Christian, Amelie Seyfried et al.
A Comparison of Elementary Baselines for BabyLM
Rareș Păpușoi, Sergiu Nisioi