CL-HOI: Cross-Level Human-Object Interaction Distillation from Vision Large Language Models
This work addresses the need for efficient, annotation-free HOI detection in computer vision, though it is incremental as it builds on existing distillation and VLLM techniques.
The paper tackles the problem of human-object interaction detection without manual annotations by distilling instance-level knowledge from vision large language models, achieving superior performance on HICO-DET and V-COCO datasets compared to existing weakly supervised and supervised methods.
Human-object interaction (HOI) detection has seen advancements with Vision Language Models (VLMs), but these methods often depend on extensive manual annotations. Vision Large Language Models (VLLMs) can inherently recognize and reason about interactions at the image level but are computationally heavy and not designed for instance-level HOI detection. To overcome these limitations, we propose a Cross-Level HOI distillation (CL-HOI) framework, which distills instance-level HOIs from VLLMs image-level understanding without the need for manual annotations. Our approach involves two stages: context distillation, where a Visual Linguistic Translator (VLT) converts visual information into linguistic form, and interaction distillation, where an Interaction Cognition Network (ICN) reasons about spatial, visual, and context relations. We design contrastive distillation losses to transfer image-level context and interaction knowledge from the teacher to the student model, enabling instance-level HOI detection. Evaluations on HICO-DET and V-COCO datasets demonstrate that our CL-HOI surpasses existing weakly supervised methods and VLLM supervised methods, showing its efficacy in detecting HOIs without manual labels.