An analysis of HOI: using a training-free method with multimodal visual foundation models when only the test set is available, without the training set
This addresses a novel scenario in HOI recognition for researchers, but it is incremental as it builds on existing foundation models without introducing new methods.
The study tackled the problem of performing Human-Object Interaction (HOI) recognition using only a test dataset without training data, employing multimodal visual foundation models in a training-free manner. It found that the open vocabulary capabilities of these models are not fully realized, with conclusions drawn from experiments using grounding truth and random arbitrary combinations.
Human-Object Interaction (HOI) aims to identify the pairs of humans and objects in images and to recognize their relationships, ultimately forming $\langle human, object, verb \rangle$ triplets. Under default settings, HOI performance is nearly saturated, with many studies focusing on long-tail distribution and zero-shot/few-shot scenarios. Let us consider an intriguing problem:``What if there is only test dataset without training dataset, using multimodal visual foundation model in a training-free manner? '' This study uses two experimental settings: grounding truth and random arbitrary combinations. We get some interesting conclusion and find that the open vocabulary capabilities of the multimodal visual foundation model are not yet fully realized. Additionally, replacing the feature extraction with grounding DINO further confirms these findings.