Learning Human-Human Interactions in Images from Weak Textual Supervision
This addresses the challenge of modeling the unlimited space of human interactions in images for applications in computer vision and AI, though it is incremental in leveraging existing distillation techniques.
The paper tackles the problem of learning diverse human-human interactions from still images by treating them as free text instead of categories, using weak textual supervision from synthetic captions generated by a large language model. The result is a captioning model that outperforms state-of-the-art methods on this task, as measured by textual and semantic metrics.
Interactions between humans are diverse and context-dependent, but previous works have treated them as categorical, disregarding the heavy tail of possible interactions. We propose a new paradigm of learning human-human interactions as free text from a single still image, allowing for flexibility in modeling the unlimited space of situations and relationships between people. To overcome the absence of data labelled specifically for this task, we use knowledge distillation applied to synthetic caption data produced by a large language model without explicit supervision. We show that the pseudo-labels produced by this procedure can be used to train a captioning model to effectively understand human-human interactions in images, as measured by a variety of metrics that measure textual and semantic faithfulness and factual groundedness of our predictions. We further show that our approach outperforms SOTA image captioning and situation recognition models on this task. We will release our code and pseudo-labels along with Waldo and Wenda, a manually-curated test set for still image human-human interaction understanding.