A Modular Approach to Conversational Object Grounding in 3D Scenes Using VLMs and Prompt Chaining
Abstract
AbstractWe introduce Conversational Object Grounding, a new task in which an agent must identify a target object in a complex 3D scene by engaging in natural language dialogue with a user who knows the target. It extends multimodal reference resolution on a single referring expression to conversations over multiple turns, combining it with clarification question generation and answer understanding to resolve ambiguities. For this task, we propose a zero-shot modular approach that combines the reference resolution, object captioning, and dialogue capabilities of foundation models, using prompt chaining in conjunction with an image-grounded scene graph. Evaluations on a new benchmark constructed with 3D scenes from ScanNet show that the proposed modular approach outperforms end-to-end baselines in terms of both grounding performance (up to 13% improvement) and image processing cost (up to 63% reduction).