Do Language Models Understand Measurements?
This addresses a gap in numerical reasoning for language models, but it is incremental as it builds on existing methods with specific improvements.
The study tackled the problem of pre-trained language models' inability to reason over measurements, finding that training on measurement-rich data improves performance and proposing an embedding strategy that significantly enhances probing task results.
Recent success of pre-trained language models (PLMs) has stimulated interest in their ability to understand and work with numbers. Yet, the numerical reasoning over measurements has not been formally studied despite their importance. In this study, we show that PLMs lack the capability required for reasoning over measurements. Furthermore, we find that a language model trained on a measurement-rich corpus shows better performance on understanding measurements. We propose a simple embedding strategy to better distinguish between numbers and units, which leads to a significant improvement in the probing tasks.