Pretraining

Balaur: Language Model Pretraining with Lexical Semantic Relations EMNLP 2023 CLMSM: A Multi-Task Learning Framework for Pre-training on Procedural Text EMNLP 2023 Continual Pre-training of Language Models ICLR 2023 DarkBERT: A Language Model for the Dark Side of the Internet ACL 2023 DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharing ICLR 2023 DLAMA: A Framework for Curating Culturally Diverse Facts for Probing the Knowledge of Pretrained Language Models ACL 2023 DocSplit: Simple Contrastive Pretraining for Large Document Embeddings EMNLP 2023 DrBERT: A Robust Pre-trained Model in French for Biomedical and Clinical domains ACL 2023 Encoder and Decoder, Not One Less for Pre-trained Language Model Sponsored NMT ACL 2023 ESCOXLM-R: Multilingual Taxonomy-driven Pre-training for the Job Market Domain ACL 2023 Give Me the Facts! A Survey on Factual Knowledge Probing in Pre-trained Language Models EMNLP 2023 Hierarchical Pretraining on Multimodal Electronic Health Records EMNLP 2023 IDOL: Indicator-oriented Logic Pre-training for Logical Reasoning ACL 2023 Improving Syntactic Probing Correctness and Robustness with Control Tasks ACL 2023 Knowledge-Selective Pretraining for Attribute Value Extraction EMNLP 2023 Masked Latent Semantic Modeling: an Efficient Pre-training Alternative to Masked Language Modeling ACL 2023 PairSpanBERT: An Enhanced Language Model for Bridging Resolution ACL 2023 Pre-training Multi-party Dialogue Models with Latent Discourse Inference ACL 2023 TRIP: Accelerating Document-level Multilingual Pre-training via Triangular Document-level Pre-training on Parallel Data Triplets EMNLP 2023 UniMax: Fairer and More Effective Language Sampling for Large-Scale Multilingual Pretraining ICLR 2023 About Time: Do Transformers Learn Temporal Verbal Aspect? ACL 2022 AraBART: a Pretrained Arabic Sequence-to-Sequence Model for Abstractive Summarization EMNLP 2022 CogKTR: A Knowledge-Enhanced Text Representation Toolkit for Natural Language Understanding EMNLP 2022 G-MAP: General Memory-Augmented Pre-trained Language Model for Domain Tasks EMNLP 2022 HTLM: Hyper-Text Pre-Training and Prompting of Language Models ICLR 2022