Pretraining

Improving Relation Extraction through Syntax-induced Pre-training with Dependency Masking ACL 2022 Language Contamination Helps Explains the Cross-lingual Capabilities of English Pretrained Models EMNLP 2022 Making Pretrained Language Models Good Long-tailed Learners EMNLP 2022 Mask More and Mask Later: Efficient Pre-training of Masked Language Models by Disentangling the [MASK] Token EMNLP 2022 Pre-training Language Models with Deterministic Factual Knowledge EMNLP 2022 Pretraining Text Encoders with Adversarial Mixture of Training Signal Generators ICLR 2022 Pre-training Transformer Models with Sentence-Level Objectives for Answer Sentence Selection EMNLP 2022 SAS: Self-Augmentation Strategy for Language Model Pre-training AAAI 2022 Should We Be Pre-training? An Argument for End-task Aware Training as an Alternative ICLR 2022 SRL4E – Semantic Role Labeling for Emotions: A Unified Evaluation Framework ACL 2022 TAPEX: Table Pre-training via Learning a Neural SQL Executor ICLR 2022 TextPruner: A Model Pruning Toolkit for Pre-Trained Language Models ACL 2022 Using BERT Embeddings to Model Word Importance in Conversational Transcripts for Deaf and Hard of Hearing Users ACL 2022 What do Large Language Models Learn beyond Language? EMNLP 2022 ARMAN: Pre-training with Semantically Selecting and Reordering of Sentences for Persian Abstractive Summarization EMNLP 2021 BERT is to NLP what AlexNet is to CV: Can Pre-Trained Language Models Identify Analogies? ACL 2021 Convolutions and Self-Attention: Re-interpreting Relative Positions in Pre-trained Language Models ACL 2021 Data Efficient Masked Language Modeling for Vision and Language EMNLP 2021 DEBERTA: DECODING-ENHANCED BERT WITH DISENTANGLED ATTENTION ICLR 2021 ERNIE-Doc: A Retrospective Long-Document Modeling Transformer ACL 2021 GraPPa: Grammar-Augmented Pre-Training for Table Semantic Parsing ICLR 2021 nmT5 - Is parallel data still relevant for pre-training massively multilingual language models? ACL 2021 Not All Models Localize Linguistic Knowledge in the Same Place: A Layer-wise Probing on BERToids’ Representations EMNLP 2021 On the Influence of Masking Policies in Intermediate Pre-training EMNLP 2021 PMI-Masking: Principled masking of correlated spans ICLR 2021