Let's Play for Action: Recognizing Activities of Daily Living by Learning from Life Simulation Video Games
This addresses the data collection bottleneck for ADL recognition in assistive robotics and smart-homes, offering a privacy-friendly alternative, but it is incremental as it applies existing methods to a new synthetic data source.
The paper tackles the problem of recognizing Activities of Daily Living (ADL) by proposing to use life simulation video games like THE SIMS 4 to create training data, introducing the SIMS4ACTION dataset and a GamingToReal benchmark, and showing that this approach provides an inexpensive and less intrusive data source, though tasks mixing gaming and real data remain challenging.
Recognizing Activities of Daily Living (ADL) is a vital process for intelligent assistive robots, but collecting large annotated datasets requires time-consuming temporal labeling and raises privacy concerns, e.g., if the data is collected in a real household. In this work, we explore the concept of constructing training examples for ADL recognition by playing life simulation video games and introduce the SIMS4ACTION dataset created with the popular commercial game THE SIMS 4. We build Sims4Action by specifically executing actions-of-interest in a "top-down" manner, while the gaming circumstances allow us to freely switch between environments, camera angles and subject appearances. While ADL recognition on gaming data is interesting from the theoretical perspective, the key challenge arises from transferring it to the real-world applications, such as smart-homes or assistive robotics. To meet this requirement, Sims4Action is accompanied with a GamingToReal benchmark, where the models are evaluated on real videos derived from an existing ADL dataset. We integrate two modern algorithms for video-based activity recognition in our framework, revealing the value of life simulation video games as an inexpensive and far less intrusive source of training data. However, our results also indicate that tasks involving a mixture of gaming and real data are challenging, opening a new research direction. We will make our dataset publicly available at https://github.com/aroitberg/sims4action.