IllusionVQA: A Challenging Optical Illusion Dataset for Vision-Language Models

[Download Paper] [Source Code] [Webpage]

Vision-language models are often evaluated on object recognition and commonsense VQA, but those tasks assume the image is reasonable. IllusionVQA is a dataset of optical illusions and hard-to-interpret scenes with two multiple-choice tasks: comprehension and soft localization. GPT-4V, the strongest model at the time, reaches 63% on comprehension and 50% on localization with few-shot prompting, while humans are near ceiling. In-context learning and chain-of-thought can even hurt localization, and models fail to locate illusions even when the correct answer is sitting in the prompt as a few-shot example.