Pretraining

Pre-training Text-to-Text Transformers for Concept-centric Common Sense ICLR 2021 Rethinking Embedding Coupling in Pre-trained Language Models ICLR 2021 Rethinking Positional Encoding in Language Pre-training ICLR 2021 SCoRe: Pre-Training for Context Representation in Conversational Semantic Parsing ICLR 2021 Superbizarre Is Not Superb: Derivational Morphology Improves BERT’s Interpretation of Complex Words ACL 2021 Syntax-Enhanced Pre-trained Model ACL 2021 Taking Notes on the Fly Helps Language Pre-Training ICLR 2021 Analyzing Individual Neurons in Pre-trained Language Models EMNLP 2020 Cost-effective Selection of Pretraining Data: A Case Study of Pretraining BERT on Social Media EMNLP 2020 Do Language Embeddings capture Scales? EMNLP 2020 ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators ICLR 2020 Getting the ##life out of living: How Adequate Are Word-Pieces for Modelling Complex Morphology? ACL 2020 MPNet: Masked and Permuted Pre-training for Language Understanding NeurIPS 2020 Pretrained Encyclopedia: Weakly Supervised Knowledge-Pretrained Language Model ICLR 2020 Revisiting Pre-Trained Models for Chinese Natural Language Processing EMNLP 2020 SciWING– A Software Toolkit for Scientific Document Processing EMNLP 2020 StructBERT: Incorporating Language Structures into Pre-training for Deep Language Understanding ICLR 2020 TaPas: Weakly Supervised Table Parsing via Pre-training ACL 2020 Which *BERT? A Survey Organizing Contextualized Encoders EMNLP 2020 BERT Rediscovers the Classical NLP Pipeline ACL 2019 Explicit Cross-lingual Pre-training for Unsupervised Machine Translation EMNLP 2019 Knowledge Enhanced Contextual Word Representations EMNLP 2019 Multi-News: A Large-Scale Multi-Document Summarization Dataset and Abstractive Hierarchical Model ACL 2019 SciBERT: A Pretrained Language Model for Scientific Text EMNLP 2019 UER: An Open-Source Toolkit for Pre-training Models EMNLP 2019