language model pretraining

RetroMAE: Pre-Training Retrieval-oriented Language Models Via Masked Auto-Encoder EMNLP 2022 Self-Supervised Knowledge Assimilation for Expert-Layman Text Style Transfer AAAI 2022 TAPEX: Table Pre-training via Learning a Neural SQL Executor ICLR 2022 Teaching Broad Reasoning Skills for Multi-Step QA by Generating Hard Contexts EMNLP 2022 Transformer with Memory Replay AAAI 2022 Two-Pass Low Latency End-to-End Spoken Language Understanding Interspeech 2022 Unsupervised Boundary-Aware Language Model Pretraining for Chinese Sequence Labeling EMNLP 2022 Unsupervised Corpus Aware Language Model Pre-training for Dense Passage Retrieval ACL 2022 ZusammenQA: Data Augmentation with Specialized Models for Cross-lingual Open-retrieval Question Answering System NAACL 2022 Allocating Large Vocabulary Capacity for Cross-Lingual Language Model Pre-Training EMNLP 2021 APGN: Adversarial and Parameter Generation Networks for Multi-Source Cross-Domain Dependency Parsing EMNLP 2021 BERTAC: Enhancing Transformer-based Language Models with Adversarially Pretrained Convolutional Neural Networks ACL 2021 Bilingual Alignment Pre-Training for Zero-Shot Cross-Lingual Transfer EMNLP 2021 CLIP: A Dataset for Extracting Action Items for Physicians from Hospital Discharge Notes ACL 2021 Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpus EMNLP 2021 ERNIE-M: Enhanced Multilingual Representation by Aligning Cross-lingual Semantics with Monolingual Corpora EMNLP 2021 HerBERT: Efficiently Pretrained Transformer-based Language Model for Polish EACL 2021 How to Train BERT with an Academic Budget EMNLP 2021 Improving BERT Model Using Contrastive Learning for Biomedical Relation Extraction NAACL 2021 Knowledge Graph Based Synthetic Corpus Generation for Knowledge-Enhanced Language Model Pre-training NAACL 2021 Language Model Pretraining and Transfer Learning for Very Low Resource Languages EMNLP 2021 LMSOC: An Approach for Socially Sensitive Pretraining EMNLP 2021 Meta-Learning for Few-Shot Named Entity Recognition ACL 2021 nmT5 - Is parallel data still relevant for pre-training massively multilingual language models? ACL 2021 On the Role of Corpus Ordering in Language Modeling EMNLP 2021