Nothing from Something: Can a Language Model Discover 0?
For AI researchers studying mathematical reasoning and generalization, this work shows that language models require explicit training examples to discover new mathematical concepts, with language pretraining providing a modest benefit.
The paper investigates whether language models can discover the concept of zero through out-of-distribution generalization in arithmetic tasks. It finds that GPT-2-sized models cannot generalize zero at test time, but can learn it from tens to hundreds of examples, with language pretraining reducing required examples by ~50%.
AI systems based on artificial neural networks are being developed with aspirations of pushing the boundary of human mathematical knowledge. A key question for these systems is how much they can reach beyond their training data. Mathematical discovery requires a strong form of out of distribution generalization; the ability to hypothesize genuinely new - and potentially logically more powerful - mathematical structures. It has been hypothesized that language abilities support such generalizations in human cognition. In this work, we use simple arithmetic as a case study for examining how modern AI models could expand their mathematical horizons, evaluating whether these models can independently discover the concept of "zero". We show that We show that (1) language models of a GPT-2 size are unable to perform this generalization at test time regardless of language pretraining, but (2) models can improve substantially after training on tens or hundreds of examples of zero. Additionally, we find that language pretraining reduces the number of required examples by approximately $50\%$, showing that language abilities can scaffold mathematical discovery in neural models.