HalluciDet: Hallucinating RGB Modality for Person Detection Through Privileged Information
This addresses domain adaptation for object detection in aerial imagery, specifically for IR to RGB modality shifts, but is incremental as it builds on existing image translation and detection techniques.
The paper tackles cross-modal pedestrian detection from infrared (IR) images by proposing HalluciDet, an image translation model that generates representations to reduce detection loss for an RGB detector, improving accuracy without accessing RGB data. It shows improved detection performance compared to state-of-the-art methods in most cases.
A powerful way to adapt a visual recognition model to a new domain is through image translation. However, common image translation approaches only focus on generating data from the same distribution as the target domain. Given a cross-modal application, such as pedestrian detection from aerial images, with a considerable shift in data distribution between infrared (IR) to visible (RGB) images, a translation focused on generation might lead to poor performance as the loss focuses on irrelevant details for the task. In this paper, we propose HalluciDet, an IR-RGB image translation model for object detection. Instead of focusing on reconstructing the original image on the IR modality, it seeks to reduce the detection loss of an RGB detector, and therefore avoids the need to access RGB data. This model produces a new image representation that enhances objects of interest in the scene and greatly improves detection performance. We empirically compare our approach against state-of-the-art methods for image translation and for fine-tuning on IR, and show that our HalluciDet improves detection accuracy in most cases by exploiting the privileged information encoded in a pre-trained RGB detector. Code: https://github.com/heitorrapela/HalluciDet