Agglomerative Transformer for Human-Object Interaction Detection
This addresses the problem of efficiently and accurately detecting human-object interactions in computer vision, with incremental improvements in method integration.
The paper tackles human-object interaction detection by proposing an agglomerative Transformer that dynamically clusters patch tokens to extract instance-level cues, achieving a new state-of-the-art performance of 36.75 mAP on HICO-Det and improving efficiency with 8.5% reduced GFLOPs and 36% higher FPS.
We propose an agglomerative Transformer (AGER) that enables Transformer-based human-object interaction (HOI) detectors to flexibly exploit extra instance-level cues in a single-stage and end-to-end manner for the first time. AGER acquires instance tokens by dynamically clustering patch tokens and aligning cluster centers to instances with textual guidance, thus enjoying two benefits: 1) Integrality: each instance token is encouraged to contain all discriminative feature regions of an instance, which demonstrates a significant improvement in the extraction of different instance-level cues and subsequently leads to a new state-of-the-art performance of HOI detection with 36.75 mAP on HICO-Det. 2) Efficiency: the dynamical clustering mechanism allows AGER to generate instance tokens jointly with the feature learning of the Transformer encoder, eliminating the need of an additional object detector or instance decoder in prior methods, thus allowing the extraction of desirable extra cues for HOI detection in a single-stage and end-to-end pipeline. Concretely, AGER reduces GFLOPs by 8.5% and improves FPS by 36%, even compared to a vanilla DETR-like pipeline without extra cue extraction.