Beyond Redundancy: Information-Conserving Semantic Packaging for Vision-Language Models
Abstract
AbstractVision-Language Models (VLMs) achieve strong multimodal understanding but suffer from high inference costs due to redundant vision tokens. Existing hard-pruning methods discard low-scoring tokens directly, inevitably sacrificing fine-grained visual context. We propose SPack, a training-free token compression method that reframes pruning as information-energy redistribution rather than deletion. Instead of discarding pruned tokens, SPack transfers their semantic mass to retained tokens, enforcing a global principle we term Information Energy Conservation (IEC): the total information contained in redundant tokens is strictly conserved and fully transferred as information gain to the retained components. To realize this conservation, we introduce Optimal Transport (OT) theory, which treats removed energy as mass density and retained tokens as receiver slots. Solving the optimal transport plan under the Wasserstein distance yields a minimum-cost mapping that precisely transfers geometric structure and higher-order moments according to semantic affinity. Concretely, we estimate token importance via relation attention, semantic affinity, and value-space energy, then adaptively route pruned tokens to multiple retained tokens using OT, producing hyper-dense tokens that preserve both local semantics and global context. Extensive experiments demonstrate that SPack consistently achieves a superior accuracy-efficiency trade-off compared to state-of-the-art methods. The code will be released upon acceptance.