Evaluating Pragmatic Abilities of Image Captioners on A3DS
This provides a dataset for evaluating pragmatic qualities in image captioning, addressing a gap in human-collected data, but it is incremental as it builds on existing captioning methods.
The authors tackled the challenge of evaluating pragmatic abilities in image captioners by introducing the A3DS dataset with over nine million annotations, and showed that a fine-tuned model develops human-like patterns such as informativity and brevity.
Evaluating grounded neural language model performance with respect to pragmatic qualities like the trade off between truthfulness, contrastivity and overinformativity of generated utterances remains a challenge in absence of data collected from humans. To enable such evaluation, we present a novel open source image-text dataset "Annotated 3D Shapes" (A3DS) comprising over nine million exhaustive natural language annotations and over 12 million variable-granularity captions for the 480,000 images provided by Burges & Kim (2018). We showcase the evaluation of pragmatic abilities developed by a task-neutral image captioner fine-tuned in a multi-agent communication setting to produce contrastive captions. The evaluation is enabled by the dataset because the exhaustive annotations allow to quantify the presence of contrastive features in the model's generations. We show that the model develops human-like patterns (informativity, brevity, over-informativity for specific features (e.g., shape, color biases)).