Illumination-aware Faster R-CNN for Robust Multispectral Pedestrian Detection
This work addresses robust pedestrian detection for autonomous vehicles or surveillance systems, offering an incremental improvement by optimizing fusion strategies for multispectral data.
The paper tackles the problem of effectively fusing color and thermal modalities for pedestrian detection under varying illumination conditions, proposing an Illumination-aware Faster R-CNN that adaptively merges sub-networks based on illumination measures, achieving state-of-the-art results on the KAIST benchmark.
Multispectral images of color-thermal pairs have shown more effective than a single color channel for pedestrian detection, especially under challenging illumination conditions. However, there is still a lack of studies on how to fuse the two modalities effectively. In this paper, we deeply compare six different convolutional network fusion architectures and analyse their adaptations, enabling a vanilla architecture to obtain detection performances comparable to the state-of-the-art results. Further, we discover that pedestrian detection confidences from color or thermal images are correlated with illumination conditions. With this in mind, we propose an Illumination-aware Faster R-CNN (IAF RCNN). Specifically, an Illumination-aware Network is introduced to give an illumination measure of the input image. Then we adaptively merge color and thermal sub-networks via a gate function defined over the illumination value. The experimental results on KAIST Multispectral Pedestrian Benchmark validate the effectiveness of the proposed IAF R-CNN.