B-RIGHT: Benchmark Re-evaluation for Integrity in Generalized Human-Object Interaction Testing
This addresses evaluation integrity for researchers in computer vision, though it is incremental as it focuses on dataset re-evaluation rather than a new method.
The paper tackles the problem of unreliable evaluation in human-object interaction (HOI) benchmarks due to class imbalance and varying dataset sizes, by proposing B-RIGHT, a new class-balanced dataset that reduces score variance and changes model performance rankings compared to HICO-DET.
Human-object interaction (HOI) is an essential problem in artificial intelligence (AI) which aims to understand the visual world that involves complex relationships between humans and objects. However, current benchmarks such as HICO-DET face the following limitations: (1) severe class imbalance and (2) varying number of train and test sets for certain classes. These issues can potentially lead to either inflation or deflation of model performance during evaluation, ultimately undermining the reliability of evaluation scores. In this paper, we propose a systematic approach to develop a new class-balanced dataset, Benchmark Re-evaluation for Integrity in Generalized Human-object Interaction Testing (B-RIGHT), that addresses these imbalanced problems. B-RIGHT achieves class balance by leveraging balancing algorithm and automated generation-and-filtering processes, ensuring an equal number of instances for each HOI class. Furthermore, we design a balanced zero-shot test set to systematically evaluate models on unseen scenario. Re-evaluating existing models using B-RIGHT reveals substantial the reduction of score variance and changes in performance rankings compared to conventional HICO-DET. Our experiments demonstrate that evaluation under balanced conditions ensure more reliable and fair model comparisons.