A Study of Failure Modes in Two-Stage Human-Object Interaction Detection
For researchers in HOI detection, this study provides insights into model limitations and failure patterns, but it is an incremental analysis without proposing new methods or achieving quantitative improvements.
This paper analyzes failure modes of two-stage HOI models by decomposing detection into interpretable perspectives and curating a subset of images from an existing dataset. It finds that high overall benchmark performance does not necessarily reflect robust visual reasoning, especially in complex scenes with multiple people and rare interactions.
Human-object interaction (HOI) detection aims to detect interactions between humans and objects in images. While recent advances have improved performance on existing benchmarks, their evaluations mainly focus on overall prediction accuracy and provide limited insight into the underlying causes of model failures. In particular, modern models often struggle in complex scenes involving multiple people and rare interaction combinations. In this work, we present a study to better understand the failure modes of two-stage HOI models, which form the basis of many current HOI detection approaches. Rather than constructing a large-scale benchmark, we instead decompose HOI detection into multiple interpretable perspectives and analyze model behavior across these dimensions to study different types of failure patterns. We curate a subset of images from an existing HOI dataset organized by human-object-interaction configurations (e.g., multi-person interactions and object sharing), and analyze model behavior under these configurations to examine different failure modes. This design allows us to analyze how these HOI models behave under different scene compositions and why their predictions fail. Importantly, high overall benchmark performance does not necessarily reflect robust visual reasoning about human-object relationships. We hope that this study can provide useful insights into the limitations of HOI models and offer observations for future research in this area.