conftrace_

sparsely activated model

2 papers

Explore in graph

Also known as

SAM

Co-occurring keywords · papers mentioning both

checkpoint reuse (1) model upcycling (1) dense checkpoint (1) parameter efficiency (1) sparse training (1) mixture of expert (1) consistency regularization (1) machine translation (1)

Papers

Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpoints ICLR 2023 Taming Sparsely Activated Transformer with Stochastic Experts ICLR 2022