Do Large Language Models Truly Understand Geometric Structures?
This work addresses a specific challenge in evaluating LLMs' geometric abilities for AI research, but it is incremental as it builds on existing methods like chain-of-thought.
The authors tackled the problem of evaluating whether large language models (LLMs) truly understand geometric structures by introducing the GeomRel dataset to isolate geometric relationship identification, and they proposed the Geometry Chain-of-Thought (GeoCoT) method, which resulted in significant performance improvements.
Geometric ability is a significant challenge for large language models (LLMs) due to the need for advanced spatial comprehension and abstract thinking. Existing datasets primarily evaluate LLMs on their final answers, but they cannot truly measure their true understanding of geometric structures, as LLMs can arrive at correct answers by coincidence. To fill this gap, we introduce the GeomRel dataset, designed to evaluate LLMs' understanding of geometric structures by isolating the core step of geometric relationship identification in problem-solving. Using this benchmark, we conduct thorough evaluations of diverse LLMs and identify key limitations in understanding geometric structures. We further propose the Geometry Chain-of-Thought (GeoCoT) method, which enhances LLMs' ability to identify geometric relationships, resulting in significant performance improvements.