model pretraining

Hyper-Connections ICLR 2025 Masking meets Supervision: A Strong Learning Alliance CVPR 2025 Overlapping Context with Variable-Length Stride Increases Diversity when Training Large Language Model for Code ACL 2025 Parameter and Memory Efficient Pretraining via Low-rank Riemannian Optimization ICLR 2025 TorchTitan: One-stop PyTorch native solution for production ready LLM pretraining ICLR 2025 Towards Pre-training an Effective Respiratory Audio Foundation Model Interspeech 2025 An Empirical Comparison of Vocabulary Expansion and Initialization Approaches For Language Models CoNLL 2024 Characterization of Large Language Model Development in the Datacenter NSDI 2024 DeepSpeed Data Efficiency: Improving Deep Learning Model Quality and Training Efficiency via Efficient Data Sampling and Routing AAAI 2024 Exploring Quantization for Efficient Pre-Training of Transformer Language Models EMNLP 2024 On the Inductive Bias of Stacking Towards Improving Reasoning NeurIPS 2024 ParameterNet: Parameters Are All You Need for Large-scale Visual Pretraining of Mobile Networks CVPR 2024 Pre-training Differentially Private Models with Limited Public Data NeurIPS 2024 Scaling Parameter-Constrained Language Models with Quality Data EMNLP 2024 SFOD: Spiking Fusion Object Detector CVPR 2024 AdaTranS: Adapting with Boundary-based Shrinking for End-to-End Speech Translation EMNLP 2023 EHRSHOT: An EHR Benchmark for Few-Shot Evaluation of Foundation Models NeurIPS 2023 Gold-YOLO: Efficient Object Detector via Gather-and-Distribute Mechanism NeurIPS 2023 Maximizing Communication Efficiency for Large-scale Training via 0/1 Adam ICLR 2023 nanoT5: Fast & Simple Pre-training and Fine-tuning of T5 Models with Limited Resources EMNLP 2023 Pre-training Multi-party Dialogue Models with Latent Discourse Inference ACL 2023 Recipes for Sequential Pre-training of Multilingual Encoder and Seq2Seq Models ACL 2023 Reducing Barriers to Self-Supervised Learning: HuBERT Pre-training with Academic Compute Interspeech 2023 Revealing the Dark Secrets of Masked Image Modeling CVPR 2023 SpeechMatrix: A Large-Scale Mined Corpus of Multilingual Speech-to-Speech Translations ACL 2023