Randomly Sampled Language Reasoning Problems Elucidate Limitations of In-Context Learning
This work addresses the issue of LLM reliability for researchers and practitioners by highlighting limitations in in-context learning on novel problems, though it is incremental as it builds on existing theories of LLM underperformance.
The paper tackled the problem of understanding why large language models (LLMs) underperform on novel tasks by isolating novelty from complexity, using randomly sampled simple language tasks; the result was that LLMs uniformly underperformed n-gram models on these unseen tasks, both in next token prediction and chain-of-thought settings.
While LLMs have revolutionized the field of machine learning due to their high performance on a strikingly wide range of problems, they are also known to hallucinate false answers and underperform on less canonical versions of the same tasks. There are several emerging theories of LLM performance, among them that LLMs lack world modeling ability, that they have an undesirable bias towards an autoregressive prior, and that they struggle on more novel problems. The existing literature on LLM input novelty has focused on tasks of relatively high complexity, studying perturbations of canonical but complex problems. In this paper, we attempt to minimize complexity in order to isolate novelty as a factor in LLM underperformance and investigate the power of in-context-learning. To this end, we consider an extremely simple domain: next token prediction on simple language tasks. The twist is that these language tasks are wholly unseen, as they are randomly drawn from a large, parsimoniously defined set of languages arising from simple grammar rules. This experimental setup allows us to evaluate ICL independently of models' parametric knowledge. We find that LLMs uniformly underperform n-gram models on this task, both when used as next token predictors and in chain-of-thought.