Muscles in Action
This work addresses the need for richer virtual human models in sports, fitness, and AR/VR by incorporating muscle activity into computer vision, though it is an incremental step as it builds on existing motion representation methods.
The paper tackles the problem of representing internal muscle activity in human motion by introducing the Muscles in Action (MIA) dataset with 12.5 hours of synchronized video and sEMG data from 10 subjects, and learns a bidirectional model that predicts muscle activation from video and reconstructs motion from muscle activation, evaluated on in-distribution and out-of-distribution tasks.
Human motion is created by, and constrained by, our muscles. We take a first step at building computer vision methods that represent the internal muscle activity that causes motion. We present a new dataset, Muscles in Action (MIA), to learn to incorporate muscle activity into human motion representations. The dataset consists of 12.5 hours of synchronized video and surface electromyography (sEMG) data of 10 subjects performing various exercises. Using this dataset, we learn a bidirectional representation that predicts muscle activation from video, and conversely, reconstructs motion from muscle activation. We evaluate our model on in-distribution subjects and exercises, as well as on out-of-distribution subjects and exercises. We demonstrate how advances in modeling both modalities jointly can serve as conditioning for muscularly consistent motion generation. Putting muscles into computer vision systems will enable richer models of virtual humans, with applications in sports, fitness, and AR/VR.