bert pretraining

HybridBERT - Making BERT Pretraining More Efficient Through Hybrid Mixture of Attention Mechanisms NAACL 2024 Symmetric Dot-Product Attention for Efficient Training of BERT Language Models ACL 2024 BEST: BERT Pre-training for Sign Language Recognition with Coupling Tokenization AAAI 2023 CAME: Confidence-guided Adaptive Memory Efficient Optimization ACL 2023 PeCo: Perceptual Codebook for BERT Pre-training of Vision Transformers AAAI 2023 Revisiting Token Dropping Strategy in Efficient BERT Pretraining ACL 2023 BEVT: BERT Pretraining of Video Transformers CVPR 2022 Mixed-Phoneme BERT: Improving BERT with Mixed Phoneme and Sup-Phoneme Representations for Text to Speech Interspeech 2022 Token Dropping for Efficient BERT Pretraining ACL 2022 How to Train BERT with an Academic Budget EMNLP 2021 Cost-effective Selection of Pretraining Data: A Case Study of Pretraining BERT on Social Media EMNLP 2020 Entity Enhanced BERT Pre-training for Chinese NER EMNLP 2020 Large Batch Optimization for Deep Learning: Training BERT in 76 minutes ICLR 2020 Let’s Stop Incorrect Comparisons in End-to-end Relation Extraction! EMNLP 2020 On Losses for Modern Language Models EMNLP 2020 Order Matters: Semantic-Aware Neural Networks for Binary Code Similarity Detection AAAI 2020 Stance Prediction and Claim Verification: An Arabic Perspective ACL 2020 vq-wav2vec: Self-Supervised Learning of Discrete Speech Representations ICLR 2020