NYT-Connections: A Deceptively Simple Text Classification Task that Stumps System-1 Thinkers
This provides a novel tool for assessing LLM reasoning capabilities, addressing a critical gap in AI evaluation for researchers and developers.
The paper tackles the problem of evaluating deliberate reasoning in Large Language Models (LLMs) by introducing NYT-Connections, a benchmark of 358 word classification puzzles that penalize intuitive thinking, and finds that top LLMs like GPT-4 underperform humans by nearly 30%.
Large Language Models (LLMs) have shown impressive performance on various benchmarks, yet their ability to engage in deliberate reasoning remains questionable. We present NYT-Connections, a collection of 358 simple word classification puzzles derived from the New York Times Connections game. This benchmark is designed to penalize quick, intuitive "System 1" thinking, isolating fundamental reasoning skills. We evaluated six recent LLMs, a simple machine learning heuristic, and humans across three configurations: single-attempt, multiple attempts without hints, and multiple attempts with contextual hints. Our findings reveal a significant performance gap: even top-performing LLMs like GPT-4 fall short of human performance by nearly 30%. Notably, advanced prompting techniques such as Chain-of-Thought and Self-Consistency show diminishing returns as task difficulty increases. NYT-Connections uniquely combines linguistic isolation, resistance to intuitive shortcuts, and regular updates to mitigate data leakage, offering a novel tool for assessing LLM reasoning capabilities.