Learning to Evict: Policy-Driven KV Cache Optimization for Multi-Hop Multimodal Spatial Reasoning
Abstract
AbstractVisualization-of-thought (VoT) has substantially advanced multimodal spatial reasoning by interleaving visual thinking with verbal reasoning. However, multi-hop VoT reasoning introduces a fundamental scalability bottleneck: the autoregressive accumulation of interleaved visual and textual tokens causes KV cache memory to grow rapidly across successive hops. We present ScoutKV, a framework that recasts KV cache eviction as a reinforcement learning problem. ScoutKV trains a dedicated lightweight policy agent per KV head, conditioned on key–value representations, cross-hop positional encodings, and modality embeddings, to predict each token’s long-term utility across the full reasoning trajectory. Training combines offline supervised fine-tuning with future-query attention targets and online GRPO optimization with a multimodal process reward targeting low-entropy text anchors and modality-specific spatial anchors. ScoutKV consistently reduces FLOPs, latency, and peak memory while maintaining or improving accuracy across multimodal spatial reasoning benchmarks, and generalizes robustly to broader vision-language tasks.