Where Do Vision-Language Models Fail on Abstract Reasoning? An Information Processing Analysis of ARC
Abstract
AbstractCurrent vision-language models (VLMs) perform well on many multimodal benchmarks, but still struggle with ARC-style abstract reasoning, where models must infer novel grid transformations from few examples. We analyze these failures through an information-processing decomposition into encoding, transformation, and construction, comparing three controlled settings: direct image-based solving, solving from model-produced text grids, and solving from gold text grids. Across existing ARC-1 and ARC-2 together with our controllable diagnostic benchmark V-ARAOC, we observe benchmark-dependent diagnostic signatures: ARC-1 is highly sensitive to encoding quality, ARC-2 remains difficult even after transcription noise is removed, and V-ARAOC is highly sensitive to constrained output construction. These results reveal benchmark-dependent information-processing bottlenecks in VLM-based ARC solving, with different benchmarks exhibiting distinct dominant failure modes.