Recurrent Models for Situation Recognition
This work addresses situation recognition in images, which is important for computer vision applications, but it is incremental as it builds on prior CRF-based methods with a novel RNN approach.
The authors tackled the problem of predicting structured image situations (actions and noun entities) by proposing RNN models, achieving state-of-the-art accuracy on the imSitu dataset and showing that learned features improve image captioning for human-object interactions.
This work proposes Recurrent Neural Network (RNN) models to predict structured 'image situations' -- actions and noun entities fulfilling semantic roles related to the action. In contrast to prior work relying on Conditional Random Fields (CRFs), we use a specialized action prediction network followed by an RNN for noun prediction. Our system obtains state-of-the-art accuracy on the challenging recent imSitu dataset, beating CRF-based models, including ones trained with additional data. Further, we show that specialized features learned from situation prediction can be transferred to the task of image captioning to more accurately describe human-object interactions.