Lift-the-flap: what, where and when for context reasoning
This work addresses context reasoning in visual recognition, which is incremental as it builds on existing recurrent models by incorporating insights from human psychophysics.
The paper tackled the problem of visual context reasoning by identifying what image features, where they are located, and when they matter, using a lift-the-flap game where a target object is hidden. The proposed recurrent model achieved human-level accuracy in inferring the target object and exhibited human-like sampling behavior.
Context reasoning is critical in a wide variety of applications where current inputs need to be interpreted in the light of previous experience and knowledge. Both spatial and temporal contextual information play a critical role in the domain of visual recognition. Here we investigate spatial constraints (what image features provide contextual information and where they are located), and temporal constraints (when different contextual cues matter) for visual recognition. The task is to reason about the scene context and infer what a target object hidden behind a flap is in a natural image. To tackle this problem, we first describe an online human psychophysics experiment recording active sampling via mouse clicks in lift-the-flap games and identify clicking patterns and features which are diagnostic for high contextual reasoning accuracy. As a proof of the usefulness of these clicking patterns and visual features, we extend a state-of-the-art recurrent model capable of attending to salient context regions, dynamically integrating useful information, making inferences, and predicting class label for the target object over multiple clicks. The proposed model achieves human-level contextual reasoning accuracy, shares human-like sampling behavior and learns interpretable features for contextual reasoning.