Fine-grained Activity Recognition with Holistic and Pose based Features
This work addresses the ongoing debate in video activity recognition by clarifying when to use holistic vs. pose-based features, which is incremental as it builds on existing methods and datasets.
The paper investigates the factors influencing the performance of holistic and pose-based methods for fine-grained human activity recognition in video, finding they are highly complementary with performance varying significantly by activity, and that combining both approaches yields the best results.
Holistic methods based on dense trajectories are currently the de facto standard for recognition of human activities in video. Whether holistic representations will sustain or will be superseded by higher level video encoding in terms of body pose and motion is the subject of an ongoing debate. In this paper we aim to clarify the underlying factors responsible for good performance of holistic and pose-based representations. To that end we build on our recent dataset leveraging the existing taxonomy of human activities. This dataset includes 24,920 video snippets covering 410 human activities in total. Our analysis reveals that holistic and pose-based methods are highly complementary, and their performance varies significantly depending on the activity. We find that holistic methods are mostly affected by the number and speed of trajectories, whereas pose-based methods are mostly influenced by viewpoint of the person. We observe striking performance differences across activities: for certain activities results with pose-based features are more than twice as accurate compared to holistic features, and vice versa. The best performing approach in our comparison is based on the combination of holistic and pose-based approaches, which again underlines their complementarity.