Can Deep Learning Recognize Subtle Human Activities?
This work addresses the issue of overestimating deep learning performance for computer vision researchers, highlighting a critical gap in model generalization.
The paper tackles the problem of deep learning models performing poorly on subtle human activity recognition due to dataset biases, showing that state-of-the-art models achieve accuracies of 61.7% to 76.8% on tasks like drinking and reading, while humans score above 90%.
Deep Learning has driven recent and exciting progress in computer vision, instilling the belief that these algorithms could solve any visual task. Yet, datasets commonly used to train and test computer vision algorithms have pervasive confounding factors. Such biases make it difficult to truly estimate the performance of those algorithms and how well computer vision models can extrapolate outside the distribution in which they were trained. In this work, we propose a new action classification challenge that is performed well by humans, but poorly by state-of-the-art Deep Learning models. As a proof-of-principle, we consider three exemplary tasks: drinking, reading, and sitting. The best accuracies reached using state-of-the-art computer vision models were 61.7%, 62.8%, and 76.8%, respectively, while human participants scored above 90% accuracy on the three tasks. We propose a rigorous method to reduce confounds when creating datasets, and when comparing human versus computer vision performance. Source code and datasets are publicly available.