Enigme: Generative Text Puzzles for Evaluating Reasoning in Language Models
This provides a tool for researchers to assess reasoning capabilities in AI systems, though it appears incremental as it builds on existing evaluation paradigms.
The authors tackled the problem of evaluating reasoning limitations in transformer-decoder language models by designing text-based puzzles that probe architectural constraints, resulting in an open-source library called enigme for generating these evaluation tasks.
Transformer-decoder language models are a core innovation in text based generative artificial intelligence. These models are being deployed as general-purpose intelligence systems in many applications. Central to their utility is the capacity to understand natural language commands and exploit the reasoning embedded in human text corpora to apply some form of reasoning process to a wide variety of novel tasks. To understand the limitations of this approach to generating reasoning we argue that we need to consider the architectural constraints of these systems. Consideration of the latent variable structure of transformer-decoder models allows us to design reasoning tasks that should probe the boundary of their capacity to reason. We present enigme, an open-source library for generating text-based puzzles to be used in training and evaluating reasoning skills within transformer-decoder models and future AI architectures.