Do Language Embeddings Capture Scales?
This addresses a gap in understanding language models' capabilities for scalar reasoning, which is incremental but relevant for AI and NLP researchers.
The study investigated whether pretrained language models capture scalar magnitude knowledge, finding they possess significant but insufficient information for general common-sense reasoning, with a simple number canonicalization method notably improving results.
Pretrained Language Models (LMs) have been shown to possess significant linguistic, common sense, and factual knowledge. One form of knowledge that has not been studied yet in this context is information about the scalar magnitudes of objects. We show that pretrained language models capture a significant amount of this information but are short of the capability required for general common-sense reasoning. We identify contextual information in pre-training and numeracy as two key factors affecting their performance and show that a simple method of canonicalizing numbers can have a significant effect on the results.