Synthetic Human Action Video Data Generation with Pose Transfer
This work addresses the problem of generating realistic synthetic video data for human motion tasks like sign language translation and gesture recognition, which is incremental as it builds on pose transfer techniques.
The paper tackles the problem of synthetic human action video data generation, which often suffers from uncanny features that limit its training effectiveness, by proposing a method using pose transfer with controllable 3D Gaussian avatar models. The result shows improved performance in action recognition tasks on datasets like Toyota Smarthome and NTU RGB+D, and the method effectively scales few-shot datasets to address underrepresented groups.
In video understanding tasks, particularly those involving human motion, synthetic data generation often suffers from uncanny features, diminishing its effectiveness for training. Tasks such as sign language translation, gesture recognition, and human motion understanding in autonomous driving have thus been unable to exploit the full potential of synthetic data. This paper proposes a method for generating synthetic human action video data using pose transfer (specifically, controllable 3D Gaussian avatar models). We evaluate this method on the Toyota Smarthome and NTU RGB+D datasets and show that it improves performance in action recognition tasks. Moreover, we demonstrate that the method can effectively scale few-shot datasets, making up for groups underrepresented in the real training data and adding diverse backgrounds. We open-source the method along with RANDOM People, a dataset with videos and avatars of novel human identities for pose transfer crowd-sourced from the internet.