Three-Stream Fusion Network for First-Person Interaction Recognition
This addresses the problem of recognizing human interactions from a first-person viewpoint for applications like wearable cameras, but it is incremental as it builds on existing multi-stream architectures.
The paper tackled first-person interaction recognition by proposing a three-stream fusion network that captures target appearance, target motion, and camera ego-motion, and it outperformed all competing methods on benchmark datasets in videos with significant camera movement.
First-person interaction recognition is a challenging task because of unstable video conditions resulting from the camera wearer's movement. For human interaction recognition from a first-person viewpoint, this paper proposes a three-stream fusion network with two main parts: three-stream architecture and three-stream correlation fusion. Thre three-stream architecture captures the characteristics of the target appearance, target motion, and camera ego-motion. Meanwhile the three-stream correlation fusion combines the feature map of each of the three streams to consider the correlations among the target appearance, target motion and camera ego-motion. The fused feature vector is robust to the camera movement and compensates for the noise of the camera ego-motion. Short-term intervals are modeled using the fused feature vector, and a long short-term memory(LSTM) model considers the temporal dynamics of the video. We evaluated the proposed method on two-public benchmark datasets to validate the effectiveness of our approach. The experimental results show that the proposed fusion method successfully generated a discriminative feature vector, and our network outperformed all competing activity recognition methods in first-person videos where considerable camera ego-motion occurs.