conftrace_

pretraining efficiency

8 papers

Explore in graph

Co-occurring keywords · papers mentioning both

language model pretraining (2) masked language modeling (2) pre-trained language model (2) language model (2) attention mechanism (2) self-supervised learning (2) unlimited context length (1) mask ratio (1) difficulty-flatten loss (1) decoder masking (1)

Papers

Efficient MAE Towards Large-Scale Vision Transformers WACV 2024 Megalodon: Efficient LLM Pretraining and Inference with Unlimited Context Length NeurIPS 2024 SoftDedup: an Efficient Data Reweighting Method for Speeding Up Language Model Pre-training ACL 2024 MosaicBERT: A Bidirectional Encoder Optimized for Fast Pretraining NeurIPS 2023 PEER: Pre-training ELECTRA Extended by Ranking ACL 2023 Revisiting Token Dropping Strategy in Efficient BERT Pretraining ACL 2023 Efficient Pre-training of Masked Language Model via Concept-based Curriculum Masking EMNLP 2022 Instance Regularization for Discriminative Language Model Pre-training EMNLP 2022