Think, Act, Build: An Agentic Framework with Vision Language Models for Zero-Shot 3D Visual Grounding
Abstract
Abstract3D Visual Grounding (3D-VG) localizes objects in 3D scenes via natural language descriptions. Current zero-shot Vision-Language Models (VLMs) typically rely on preprocessed 3D point clouds and pretrained 3D detectors, reducing grounding to static proposal matching. To address this, we propose “Think, Act, Build (TAB)”, a dynamic agentic framework that reformulates 3D-VG as a 2D-to-3D reconstruction paradigm operating directly on raw RGB-D streams. TAB decouples the task by leveraging 2D VLMs for complex spatial semantics and multi-view geometry for 3D structure instantiation. Guided by a specialized skill, our VLM agent dynamically invokes visual tools to track targets across 2D frames. To overcome multi-view coverage deficits inherent in strict VLM semantic tracking, we introduce Semantic-Anchored Geometric Expansion. This mechanism anchors the target in a reference video clip and uses multi-view geometry to propagate its spatial location across unobserved frames, mapping 2D visual cues to 3D coordinates. Additionally, we manually refine existing benchmarks to eliminate reference ambiguity and category errors. Extensive experiments on ScanRefer and Nr3D demonstrate that TAB, relying entirely on open-source models, significantly outperforms previous zero-shot methods and even supervised baselines.