Pretraining

ArabicTransformer: Efficient Large Arabic Language Model with Funnel Transformer and ELECTRA Objective EMNLP 2021 Are Pretrained Convolutions Better than Pretrained Transformers? ACL 2021 ARMAN: Pre-training with Semantically Selecting and Reordering of Sentences for Persian Abstractive Summarization EMNLP 2021 A Simple Approach for Handling Out-of-Vocabulary Identifiers in Deep Learning for Source Code NAACL 2021 A Study into Pre-Training Strategies for Spoken Language Understanding on Dysarthric Speech Interspeech 2021 A Thorough Evaluation of Task-Specific Pretraining for Summarization EMNLP 2021 Attending Self-Attention: A Case Study of Visually Grounded Supervision in Vision-and-Language Transformers ACL 2021 AttentionRank: Unsupervised Keyphrase Extraction using Self and Cross Attentions EMNLP 2021 AuGPT: Auxiliary Tasks and Data Augmentation for End-To-End Dialogue with Pre-Trained Language Models EMNLP 2021 Batavia asked for advice. Pretrained language models for Named Entity Recognition in historical texts. EMNLP 2021 BennettNLP at SemEval-2021 Task 5: Toxic Spans Detection using Stacked Embedding Powered Toxic Entity Recognizer SemEval 2021 BERTAC: Enhancing Transformer-based Language Models with Adversarially Pretrained Convolutional Neural Networks ACL 2021 BERT & Family Eat Word Salad: Experiments with Text Understanding AAAI 2021 BERT is to NLP what AlexNet is to CV: Can Pre-Trained Language Models Identify Analogies? ACL 2021 BERT might be Overkill: A Tiny but Effective Biomedical Entity Linker based on Residual Convolutional Neural Networks EMNLP 2021 BERTologiCoMix: How does Code-Mixing interact with Multilingual BERT? EACL 2021 Bilingual Alignment Pre-Training for Zero-Shot Cross-Lingual Transfer EMNLP 2021 Binaural Audio-Visual Localization AAAI 2021 Boundary-Sensitive Pre-Training for Temporal Localization in Videos ICCV 2021 Can Transformer Models Measure Coherence In Text: Re-Thinking the Shuffle Test ACL 2021 Char2Subword: Extending the Subword Embedding Space Using Robust Character Compositionality EMNLP 2021 ChineseBERT: Chinese Pretraining Enhanced by Glyph and Pinyin Information ACL 2021 COCO-LM: Correcting and Contrasting Text Sequences for Language Model Pretraining NeurIPS 2021 CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation EMNLP 2021 Composed Fine-Tuning: Freezing Pre-Trained Denoising Autoencoders for Improved Generalization ICML 2021