Skip to content
conftrace_
2026 EMNLP EMNLP 2026

Can Vision-Language Models Solve the Shell Game?

AbstractVisual entity tracking is an innate cognitive ability in humans, yet it remains a critical bottleneck for Vision-Language Models (VLMs). This deficit is often obscured by visual shortcuts in existing video benchmarks. We introduce VET-Bench, a synthetic diagnostic testbed featuring visually identical objects that necessitate tracking exclusively through spatiotemporal continuity. Our experiments reveal that current state-of-the-art VLMs perform at or near chance level on VET-Bench. We provide a theoretical analysis drawing connections to the state-tracking problem, suggesting that fixed-depth transformer-based VLMs are fundamentally limited in tracking indistinguishable objects without intermediate computation due to expressivity constraints. To address this, we propose Spatiotemporal Grounded Chain-of-Thought (SGCoT): generating object trajectories as explicit intermediate states. Leveraging Molmo2’s object-tracking capability, we elicit SGCoT reasoning by fine-tuning on synthesized text-only data. Our method achieves state-of-the-art accuracy exceeding 90% on VET-Bench, demonstrating that VLMs can reliably solve the video shell-game task end-to-end without external tools. Our code and data are available at https://vetbench.github.io.

More from EMNLP 2026

All EMNLP 2026 papers →