Pragmatically Informative Color Generation by Grounding Contextual Modifiers
This work addresses the challenge of fine-grained natural language understanding for color generation, which is incremental as it builds on existing grounding and pragmatics approaches.
The paper tackles the problem of generating colors from contextual modifiers like 'bluey green' by proposing a computational pragmatics model that frames the task as a recursive speaker-listener game, resulting in absolute performance increases of 98% for unseen reference colors and 40% for unseen colors and modifiers compared to state-of-the-art deep learning models.
Grounding language in contextual information is crucial for fine-grained natural language understanding. One important task that involves grounding contextual modifiers is color generation. Given a reference color "green", and a modifier "bluey", how does one generate a color that could represent "bluey green"? We propose a computational pragmatics model that formulates this color generation task as a recursive game between speakers and listeners. In our model, a pragmatic speaker reasons about the inferences that a listener would make, and thus generates a modified color that is maximally informative to help the listener recover the original referents. In this paper, we show that incorporating pragmatic information provides significant improvements in performance compared with other state-of-the-art deep learning models where pragmatic inference and flexibility in representing colors from a large continuous space are lacking. Our model has an absolute 98% increase in performance for the test cases where the reference colors are unseen during training, and an absolute 40% increase in performance for the test cases where both the reference colors and the modifiers are unseen during training.