Generics are puzzling. Can language models find the missing piece?
This work addresses the challenge of interpreting generic sentences in natural language processing, which is incremental as it applies existing language models to a new dataset and metric for a specific linguistic domain.
The study tackled the problem of understanding generic sentences by creating the ConGen dataset and introducing p-acceptability, a surprisal-based metric, to analyze their implicit quantification and context-sensitivity. The results showed that generics are more context-sensitive than determiner quantifiers, with about 20% expressing weak generalizations, and revealed how human biases in stereotypes manifest in language models.
Generic sentences express generalisations about the world without explicit quantification. Although generics are central to everyday communication, building a precise semantic framework has proven difficult, in part because speakers use generics to generalise properties with widely different statistical prevalence. In this work, we study the implicit quantification and context-sensitivity of generics by leveraging language models as models of language. We create ConGen, a dataset of 2873 naturally occurring generic and quantified sentences in context, and define p-acceptability, a metric based on surprisal that is sensitive to quantification. Our experiments show generics are more context-sensitive than determiner quantifiers and about 20% of naturally occurring generics we analyze express weak generalisations. We also explore how human biases in stereotypes can be observed in language models.