When LLMs Try to Reason: Experiments in Text and Vision-Based Abstraction

Recent experiments with large language models (LLMs), including text-based (o3-mini) and multimodal (gpt-4.1) architectures, demonstrate that while these models can perform certain pattern recognition tasks, their ability to reason abstractly from limited examples remains limited. The studies highlight that current LLMs predominantly rely on pattern matching, procedural heuristics, and symbolic shortcuts rather than developing robust, generalizable reasoning skills, especially when faced with subtle or complex abstractions in grid transformation tasks. These findings underscore the significant gap between LLMs' apparent reasoning capabilities and true abstract reasoning, even