VLMs Need Words: Vision-Language Models Ignore Visual Detail in Favor of Semantic Anchors

[Download Paper] [Source Code]

Vision-language models often fail at fine-grained visual tasks even when the needed information is present in their representations. We show that this gap comes from a language shortcut: when visual entities can be mapped to known names, VLMs skip pixel-level comparison and reason in language; when they cannot, they fall back on brittle, hallucinated descriptions. Across semantic correspondence, synthetic shape matching, and face matching, models do much better on nameable entities than unnameable ones. Teaching arbitrary names helps, and task-specific finetuning yields still stronger generalization through genuine visual comparison. The takeaway is that current failures look more like a learned training shortcut than a hard architectural limit.