Orchestrating Hidden-Intermediate Pruning-and-Distill for MoEs Slimming
Abstract
AbstractThe remarkable performance of Mixture-of-Experts (MoE) models often comes with substantial architectural scale and resource overhead. Prior structural compression mainly focuses on expert-level or FFN intermediate pruning. However, we identify an emerging architectural trend that the FFN expansion ratio in state-of-the-art MoEs has steadily decreased, making intermediate-only pruning less effective and motivating us to explore other redundancies. Motivated by this, we propose an orchestrated hidden-intermediate prune-and-distill framework that, to the best of our knowledge, is the first to jointly prune expert FFNs along both intermediate and hidden dimensions. It includes Budget-Constrained Coordinate Descent to efficiently search expert-specific pruning ratios across the two dimensions, and attention relation transfer distillation to provide stable supervision under hidden misalignment. Extensive experiments across diverse benchmarks validates that hidden-dimension pruning is a promising direction for compressing modern MoEs.