Human-Level Reasoning: A Comparative Study of Large Language Models on Logical and Abstract Reasoning
This work addresses the problem of evaluating reasoning ability in LLMs for AI researchers, but it is incremental as it benchmarks existing models without introducing new methods.
The study compared the logical and abstract reasoning skills of nine large language models against human performance on eight custom-designed questions, revealing significant differences and areas where models struggle with deduction.
Evaluating reasoning ability in Large Language Models (LLMs) is important for advancing artificial intelligence, as it transcends mere linguistic task performance. It involves understanding whether these models truly understand information, perform inferences, and are able to draw conclusions in a logical and valid way. This study compare logical and abstract reasoning skills of several LLMs - including GPT, Claude, DeepSeek, Gemini, Grok, Llama, Mistral, Perplexity, and Sabiá - using a set of eight custom-designed reasoning questions. The LLM results are benchmarked against human performance on the same tasks, revealing significant differences and indicating areas where LLMs struggle with deduction.