MSTR: Multi-Scale Transformer for End-to-End Human-Object Interaction Detection
This work addresses a domain-specific problem in computer vision for HOI detection, offering an incremental improvement over existing transformer-based methods.
The paper tackled the problem of suboptimal performance in Human-Object Interaction (HOI) detection due to single-scale feature resolution by proposing a Multi-Scale Transformer (MSTR) with novel attention modules, achieving new state-of-the-art performance on two benchmarks.
Human-Object Interaction (HOI) detection is the task of identifying a set of <human, object, interaction> triplets from an image. Recent work proposed transformer encoder-decoder architectures that successfully eliminated the need for many hand-designed components in HOI detection through end-to-end training. However, they are limited to single-scale feature resolution, providing suboptimal performance in scenes containing humans, objects and their interactions with vastly different scales and distances. To tackle this problem, we propose a Multi-Scale TRansformer (MSTR) for HOI detection powered by two novel HOI-aware deformable attention modules called Dual-Entity attention and Entity-conditioned Context attention. While existing deformable attention comes at a huge cost in HOI detection performance, our proposed attention modules of MSTR learn to effectively attend to sampling points that are essential to identify interactions. In experiments, we achieve the new state-of-the-art performance on two HOI detection benchmarks.