GPT-4o Understands Text, But Does It See Clearly? A Benchmarking Study of MFMs on Vision Tasks

Recent advancements in multimodal foundation models (MFMs) such as GPT-4o, Gemini, and Claude have demonstrated significant progress in integrating visual and language understanding, particularly in public demonstrations. While these models excel in tasks like image captioning and visual question answering (VQA), their true capacity for detailed visual comprehensionencompassing aspects like 3D perception, segmentation, and groupingremains inadequately assessed due to reliance on benchmarks primarily focused on text-based outputs and language-centric tasks. Current evaluation methods often convert visual annotations into textual prompts, which limits the ability to fairly compare MFMs