Arithmetic-Based Pretraining -- Improving Numeracy of Pretrained Language Models
This addresses the numeracy gap in language models for applications requiring numerical understanding, but it is incremental as it builds on existing pretraining methods.
The paper tackles the problem of pretrained language models performing poorly on numeracy tasks by proposing Arithmetic-Based Pretraining, which improves number representation and numeracy without architectural changes, achieving better results on datasets like DROP, InfoTabs, WikiBio, and SciGen.
State-of-the-art pretrained language models tend to perform below their capabilities when applied out-of-the-box on tasks that require understanding and working with numbers. Recent work suggests two main reasons for this: (1) popular tokenisation algorithms have limited expressiveness for numbers, and (2) common pretraining objectives do not target numeracy. Approaches that address these shortcomings usually require architectural changes or pretraining from scratch. In this paper, we propose a new extended pretraining approach called Arithmetic-Based Pretraining that jointly addresses both in one extended pretraining step without requiring architectural changes or pretraining from scratch. Arithmetic-Based Pretraining combines contrastive learning to improve the number representation, and a novel extended pretraining objective called Inferable Number Prediction Task to improve numeracy. Our experiments show the effectiveness of Arithmetic-Based Pretraining in three different tasks that require improved numeracy, i.e., reading comprehension in the DROP dataset, inference-on-tables in the InfoTabs dataset, and table-to-text generation in the WikiBio and SciGen datasets.