Logical Tasks for Measuring Extrapolation and Rule Comprehension
This work addresses the problem of improving logical reasoning in AI models for researchers and developers, but it is incremental as it builds on known limitations without presenting new experimental results or methods.
The paper identifies a limitation in large-scale models' logical reasoning abilities, particularly in arithmetic tasks, and proposes a new set of logical tasks as a broader challenge to address this issue, aiming to guide the development of inductive biases with wide impact.
Logical reasoning is essential in a variety of human activities. A representative example of a logical task is mathematics. Recent large-scale models trained on large datasets have been successful in various fields, but their reasoning ability in arithmetic tasks is limited, which we reproduce experimentally. Here, we recast this limitation as not unique to mathematics but common to tasks that require logical operations. We then propose a new set of tasks, termed logical tasks, which will be the next challenge to address. This higher point of view helps the development of inductive biases that have broad impact beyond the solution of individual tasks. We define and characterize logical tasks and discuss system requirements for their solution. Furthermore, we discuss the relevance of logical tasks to concepts such as extrapolation, explainability, and inductive bias. Finally, we provide directions for solving logical tasks.