Towards Understanding Ambiguity Resolution in Multimodal Inference of Meaning
This work addresses ambiguity resolution in foreign language learning, but it is incremental as it primarily analyzes existing data and identifies gaps without introducing new methods.
The study investigated how learners infer unfamiliar word meanings from multimodal sentence-image pairs, finding that only some intuitive features strongly correlate with performance and highlighting the need for better predictive features.
We investigate a new setting for foreign language learning, where learners infer the meaning of unfamiliar words in a multimodal context of a sentence describing a paired image. We conduct studies with human participants using different image-text pairs. We analyze the features of the data (i.e., images and texts) that make it easier for participants to infer the meaning of a masked or unfamiliar word, and what language backgrounds of the participants correlate with success. We find only some intuitive features have strong correlations with participant performance, prompting the need for further investigating of predictive features for success in these tasks. We also analyze the ability of AI systems to reason about participant performance, and discover promising future directions for improving this reasoning ability.