Skip to content
conftrace_
2026 EMNLP EMNLP 2026

Monkey Jump: MoE-Style PEFT for Efficient Multi-Task Learning

AbstractMixture-of-experts variants of parameter-efficient fine-tuning enable per-token specialization, but they introduce additional trainable routers and expert parameters, increasing memory and training costs. This undermines the core goal of parameter efficient finetuning. We propose Monkey Jump, which brings MoE-style specialization to PEFT without adding extra trainable parameters for experts and routers. Instead of introducing new PEFT adapters as experts, Monkey Jump treats the PEFT adapters already present in each Transformer block (e.g., query, key, value, up, and down projections) as implicit experts and routes tokens among them. Routing is performed via k-means clustering with EMA-updated centers—no gradients, no learned parameters. We theoretically show that token-wise routing increases expressivity and can outperform shared adapters by avoiding cancellation effects. In multi-task experiments spanning 14 text, 14 image, and 19 video benchmarks, Monkey Jump achieves competitive performance with MoE-PEFT methods while using 5.3–9.2× fewer trainable parameters than the MoE variant of the same base method, up to 48% lower memory, and 1.5–2× faster training. Monkey Jump is architecture-agnostic and can be applied to any adapter-based PEFT method.

More from EMNLP 2026

All EMNLP 2026 papers →