Beyond Transformers for Function Learning
This work addresses the gap in AI's ability to match human extrapolation in function learning, though it is incremental as it builds on existing transformer methods.
The paper tackled the problem of improving neural networks' ability to learn and extrapolate simple functions, which is a key aspect of human intelligence, by augmenting transformers with inductive biases from cognitive science, resulting in enhanced performance in extrapolation tasks.
The ability to learn and predict simple functions is a key aspect of human intelligence. Recent works have started to explore this ability using transformer architectures, however it remains unclear whether this is sufficient to recapitulate the extrapolation abilities of people in this domain. Here, we propose to address this gap by augmenting the transformer architecture with two simple inductive learning biases, that are directly adapted from recent models of abstract reasoning in cognitive science. The results we report demonstrate that these biases are helpful in the context of large neural network models, as well as shed light on the types of inductive learning biases that may contribute to human abilities in extrapolation.