Seeing Beyond the Scene: Analyzing and Mitigating Background Bias in Action Recognition
This addresses a critical issue in computer vision for improving the robustness and fairness of action recognition systems, though the improvements are incremental.
The paper tackled the problem of background bias in action recognition models, where models rely on background cues instead of human movement, and found that classification models, contrastive text-image pretrained models, and Video Large Language Models all exhibit this bias. They proposed mitigation strategies, such as incorporating segmented human input to reduce bias by 3.78% and using prompt tuning for VLLMs to increase human-focused reasoning by 9.85%.
Human action recognition models often rely on background cues rather than human movement and pose to make predictions, a behavior known as background bias. We present a systematic analysis of background bias across classification models, contrastive text-image pretrained models, and Video Large Language Models (VLLM) and find that all exhibit a strong tendency to default to background reasoning. Next, we propose mitigation strategies for classification models and show that incorporating segmented human input effectively decreases background bias by 3.78%. Finally, we explore manual and automated prompt tuning for VLLMs, demonstrating that prompt design can steer predictions towards human-focused reasoning by 9.85%.