Skip to content
conftrace_
2026 EMNLP EMNLP 2026

IWP: Token Pruning as Implicit Weight Pruning in Large Vision Language Models

AbstractLarge Vision–Language Models (LVLMs) show impressive performance across image and video understanding tasks, yet their computational cost grows rapidly with the number of visual tokens. Existing token pruning methods address this by reducing the number of visual tokens, but overlook the internal mechanism of attention. In this paper, we propose a novel training-free token pruning framework grounded in the dual-form perspective of attention. We reformulate attention as an implicit linear layer whose weight matrix is the sum of rank-1 outer products, each generated by a single token’s key–value pair. Token pruning thus reduces to identifying an optimal subset of these rank-1 updates that retains the essential information of the original dual weight matrix. Extending this perspective to standard softmax attention in LVLMs, we derive a novel importance metric quantifying both a token’s information magnitude and information duplication. To efficiently select the subset with the proposed metric, we introduce Progressive Chunked Maximal Marginal Relevance. Extensive experiments demonstrate that our method achieves a better trade-off between performance and efficiency, while providing another perspective on existing pruning approaches.

More from EMNLP 2026

All EMNLP 2026 papers →