language model pretraining

Sequence Reducible Holdout Loss for Language Model Pretraining COLING 2024 SoftDedup: an Efficient Data Reweighting Method for Speeding Up Language Model Pre-training ACL 2024 Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-training ICLR 2024 Stable Language Model Pre-training by Reducing Embedding Variability EMNLP 2024 Target-Aware Language Modeling via Granular Data Sampling EMNLP 2024 Balaur: Language Model Pretraining with Lexical Semantic Relations EMNLP 2023 Better Quality Pre-training Data and T5 Models for African Languages EMNLP 2023 ConPrompt: Pre-training a Language Model with Machine-Generated Data for Implicit Hate Speech Detection EMNLP 2023 DisorBERT: A Double Domain Adaptation Model for Detecting Signs of Mental Disorders in Social Media ACL 2023 DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretraining NeurIPS 2023 DrBERT: A Robust Pre-trained Model in French for Biomedical and Clinical domains ACL 2023 Farewell to Aimless Large-scale Pretraining: Influential Subset Selection for Language Model ACL 2023 GHisBERT – Training BERT from scratch for lexical semantic investigations across historical German language stages EMNLP 2023 GLADIS: A General and Large Acronym Disambiguation Benchmark EACL 2023 Glot500: Scaling Multilingual Corpora and Language Models to 500 Languages ACL 2023 Multi-armed bandits for resource efficient, online optimization of language model pre-training: the use case of dynamic masking ACL 2023 Not Enough Data to Pre-train Your Language Model? MT to the Rescue! ACL 2023 Patton: Language Model Pretraining on Text-Rich Networks ACL 2023 PLUE: Language Understanding Evaluation Benchmark for Privacy Policies in English ACL 2023 Pretraining Language Models with Human Preferences ICML 2023 Pretraining Language Models with Text-Attributed Heterogeneous Graphs EMNLP 2023 RetroMAE-2: Duplex Masked Auto-Encoder For Pre-Training Retrieval-Oriented Language Models ACL 2023 Self-Influence Guided Data Reweighting for Language Model Pre-training EMNLP 2023 Structure-Aware Language Model Pretraining Improves Dense Retrieval on Structured Data ACL 2023 The Web Can Be Your Oyster for Improving Language Models ACL 2023