RSA: Randomized Simulation as Augmentation for Robust Human Action Recognition
This addresses the problem of video variation for human action recognition researchers, offering an incremental improvement through data augmentation.
The paper tackles the challenge of robust human action recognition in videos by augmenting real-world training data with synthetic data generated with randomized nuisance factors, resulting in significant performance improvements for state-of-the-art I3D networks or reduced need for labeled real videos, as demonstrated on NTU RGB+D and VIRAT datasets.
Despite the rapid growth in datasets for video activity, stable robust activity recognition with neural networks remains challenging. This is in large part due to the explosion of possible variation in video -- including lighting changes, object variation, movement variation, and changes in surrounding context. An alternative is to make use of simulation data, where all of these factors can be artificially controlled. In this paper, we propose the Randomized Simulation as Augmentation (RSA) framework which augments real-world training data with synthetic data to improve the robustness of action recognition networks. We generate large-scale synthetic datasets with randomized nuisance factors. We show that training with such extra data, when appropriately constrained, can significantly improve the performance of the state-of-the-art I3D networks or, conversely, reduce the number of labeled real videos needed to achieve good performance. Experiments on two real-world datasets NTU RGB+D and VIRAT demonstrate the effectiveness of our method.