LatEval: An Interactive LLMs Evaluation Benchmark with Incomplete Information from Lateral Thinking Puzzles
This work addresses the need for better evaluation of lateral thinking in LLMs, which is crucial for developing effective AI assistants, though it is incremental as it introduces a new benchmark rather than a method.
The authors tackled the problem of evaluating lateral thinking abilities in large language models (LLMs) by proposing LatEval, an interactive benchmark based on lateral thinking puzzles, and found that nearly all LLMs, including GPT-4, struggle significantly compared to humans.
With the continuous evolution and refinement of LLMs, they are endowed with impressive logical reasoning or vertical thinking capabilities. But can they think out of the box? Do they possess proficient lateral thinking abilities? Following the setup of Lateral Thinking Puzzles, we propose a novel evaluation benchmark, LatEval, which assesses the model's lateral thinking within an interactive framework. In our benchmark, we challenge LLMs with 2 aspects: the quality of questions posed by the model and the model's capability to integrate information for problem-solving. We find that nearly all LLMs struggle with employing lateral thinking during interactions. For example, even the most advanced model, GPT-4, exhibits the advantage to some extent, yet still maintain a noticeable gap when compared to human. This evaluation benchmark provides LLMs with a highly challenging and distinctive task that is crucial to an effective AI assistant.