conftrace_

dense model

4 papers

Explore in graph

Co-occurring keywords

large language model (13587) mixture of expert (640) language model (4599) knowledge distillation (3725) model compression (3302) inference cost (28) model initialization (32) autoregressive language model (58) open-source model (27) compute budget (9)

Papers

BAM! Just Like That: Simple and Efficient Parameter Upcycling for Mixture of Experts NIPS 2024
Scaling Laws Across Model Architectures: A Comparative Analysis of Dense and MoE Models in Large Language Models EMNLP 2024
Breaking ReLU Barrier: Generalized MoEfication for Dense Pretrained Models EMNLP 2024
GPT-NeoX-20B: An Open-Source Autoregressive Language Model ACL 2022