Is "My Favorite New Movie" My Favorite Movie? Probing the Understanding of Recursive Noun Phrases
This addresses a gap in natural language understanding for AI systems, showing incremental progress by demonstrating that models can acquire this common-sense knowledge through targeted training.
The paper tackled the problem of whether language models understand the semantic properties of recursive noun phrases, such as 'my favorite new movie' not necessarily being one's favorite movie, and found that state-of-the-art Transformer models perform only around chance on the introduced Recursive Noun Phrase Challenge dataset, but this knowledge is learnable with appropriate data.
Recursive noun phrases (NPs) have interesting semantic properties. For example, "my favorite new movie" is not necessarily my favorite movie, whereas "my new favorite movie" is. This is common sense to humans, yet it is unknown whether language models have such knowledge. We introduce the Recursive Noun Phrase Challenge (RNPC), a dataset of three textual inference tasks involving textual entailment and event plausibility comparison, precisely targeting the understanding of recursive NPs. When evaluated on RNPC, state-of-the-art Transformer models only perform around chance. Still, we show that such knowledge is learnable with appropriate data. We further probe the models for relevant linguistic features that can be learned from our tasks, including modifier semantic category and modifier scope. Finally, models trained on RNPC achieve strong zero-shot performance on an extrinsic Harm Detection evaluation task, showing the usefulness of the understanding of recursive NPs in downstream applications.