STRESS-FG: Diagnosing Visual Grounding in Zero-Shot Fine-Grained Recognition
Abstract
AbstractZero-shot fine-grained recognition is a key test of visual grounding in vision-language models (VLMs), but current evaluations may conflate attribute grounding with reliance on class-label knowledge. We introduce STRESS-FG, a diagnostic benchmark spanning five fine-grained domains that systematically varies the relationship between class names and visual descriptions. The benchmark includes label-only, description-only, cue-conflict, and synthetic-label settings, as well as negated-attribute, multi-image, reference-conditioned, and part-based grounding tasks. Evaluating up to thirteen VLMs shows that label-only prompts are often surprisingly strong, indicating substantial familiarity with class-label priors. Synthetic labels can facilitate stronger attribute grounding evaluation, while familiar but incorrect class labels can suppress grounding capabilities under cue conflict. We further find that negation, multi-image matching, and reference-conditioned selection remain challenging, and part-based analyses show only partial reliance on the correct localized evidence. Finally, our binary question format reveals that models often struggle to reject fine-grained distractors, a weakness hidden by traditional Multiple Choice Question evaluations. These results suggest that standard fine-grained benchmarks may overestimate grounded visual understanding and should more explicitly separate label priors from attribute-based recognition. Code and dataset are available at: https://github.com/Ishtiaque-khan97/Stress-FG.