Geometry-Aware Recurrent Neural Networks for Active Visual Recognition
This work addresses the challenge of robust visual recognition in dynamic, occluded environments for applications like robotics and autonomous systems, representing an incremental improvement by combining geometry-aware operations with active view selection.
The paper tackles the problem of active visual recognition by integrating multiple scene views into a 3D latent feature tensor with a one-to-one mapping to physical locations, enabling object detection, segmentation, and 3D reconstruction directly from this model. It shows that the proposed geometry-aware recurrent neural networks generalize much better than geometry-unaware LSTM/GRU networks, especially in scenarios with multiple objects and cross-object occlusions.
We present recurrent geometry-aware neural networks that integrate visual information across multiple views of a scene into 3D latent feature tensors, while maintaining an one-to-one mapping between 3D physical locations in the world scene and latent feature locations. Object detection, object segmentation, and 3D reconstruction is then carried out directly using the constructed 3D feature memory, as opposed to any of the input 2D images. The proposed models are equipped with differentiable egomotion-aware feature warping and (learned) depth-aware unprojection operations to achieve geometrically consistent mapping between the features in the input frame and the constructed latent model of the scene. We empirically show the proposed model generalizes much better than geometryunaware LSTM/GRU networks, especially under the presence of multiple objects and cross-object occlusions. Combined with active view selection policies, our model learns to select informative viewpoints to integrate information from by "undoing" cross-object occlusions, seamlessly combining geometry with learning from experience.