pretraining data

Parallel Structures in Pre-training Data Yield In-Context Learning ACL 2024 Personas as a Way to Model Truthfulness in Language Models EMNLP 2024 Quantifying Contamination in Evaluating Code Generation Capabilities of Language Models ACL 2024 QuRating: Selecting High-Quality Data for Training Language Models ICML 2024 ReCaLL: Membership Inference via Relative Conditional Log-Likelihoods EMNLP 2024 Tracking the perspectives of interacting language models EMNLP 2024 What Drives Performance in Multilingual Language Models? NAACL 2024 What should Baby Models read? Exploring Sample-Efficient Data Composition on Model Performance CoNLL 2024 When Scaling Meets LLM Finetuning: The Effect of Data, Model and Finetuning Method ICLR 2024 Wikimedia data for AI: a review of Wikimedia datasets for NLP tasks and AI-assisted editing EMNLP 2024 XferBench: a Data-Driven Benchmark for Emergent Language NAACL 2024 A Data-Based Perspective on Transfer Learning CVPR 2023 Better Quality Pre-training Data and T5 Models for African Languages EMNLP 2023 Data Similarity is Not Enough to Explain Language Model Performance EMNLP 2023 Frequency Balanced Datasets Lead to Better Language Models EMNLP 2023 Large Language Models Struggle to Learn Long-Tail Knowledge ICML 2023 Understanding In-Context Learning via Supportive Pretraining Data ACL 2023 Impact of Pretraining Term Frequencies on Few-Shot Numerical Reasoning EMNLP 2022 Improved Fine-Tuning by Better Leveraging Pre-Training Data NeurIPS 2022 Pile of Law: Learning Responsible Data Filtering from the Law and a 256GB Open-Source Legal Dataset NeurIPS 2022 How much pretraining data do language models need to learn syntax? EMNLP 2021 When Do You Need Billions of Words of Pretraining Data? ACL 2021 Cost-effective Selection of Pretraining Data: A Case Study of Pretraining BERT on Social Media EMNLP 2020 Using Similarity Measures to Select Pretraining Data for NER NAACL 2019