What is an "Abstract Reasoner"? Revisiting Experiments and Arguments about Large Language Models
This work addresses the debate over LLMs' reasoning capabilities, which is important for AI researchers and practitioners, but it is incremental as it refines existing arguments with new empirical nuances.
The paper revisits claims that large language models (LLMs) are not abstract reasoners by showing that while they perform poorly zero-shot, fine-tuning a small subset of parameters enables near-perfect performance, though it does not transfer across datasets.
Recent work has argued that large language models (LLMs) are not "abstract reasoners", citing their poor zero-shot performance on a variety of challenging tasks as evidence. We revisit these experiments in order to add nuance to the claim. First, we show that while LLMs indeed perform poorly in a zero-shot setting, even tuning a small subset of parameters for input encoding can enable near-perfect performance. However, we also show that this finetuning does not necessarily transfer across datasets. We take this collection of empirical results as an invitation to (re-)open the discussion of what it means to be an "abstract reasoner", and why it matters whether LLMs fit the bill.