VISA: Video Skeleton-Aware Efficient Frame Selection for Long Video Understanding
Abstract
AbstractLong-video understanding remains challenging for multimodal large language models (MLLMs), as informative visual evidence is often sparsely distributed across many redundant frames. Although sampling more frames can improve coverage, it also increases visual encoding overhead and consumes more visual tokens, making compact frame selection essential. However, existing methods often sample candidate frames uniformly before selection, which may require decoding most frames of the video. Moreover, simply selecting frames by query relevance may overlook the global temporal structure needed for coherent video understanding. To address these limitations, we propose VISA, a VIdeo Skeleton-Aware framework for efficient long-video frame selection. VISA exploits sparse I-frame anchors from the native GOP structure to avoid dense candidate decoding and constructs a video skeleton that preserves the coarse temporal progression. Query relevance is aggregated over this skeleton to select informative anchors, followed by cross-GOP refinement to obtain final frames. Extensive experiments show that VISA consistently improves long-video understanding, achieving up to 8.8% relative gain over uniform sampling. Meanwhile, VISA reduces average end-to-end latency by 32.9%–53.6% compared with recent frame selection baselines on LongVideoBench, demonstrating both effectiveness and efficiency for long-video frame selection.