Multi-modal Crowd Counting via a Broker Modality
This work addresses the problem of accurate crowd density estimation for surveillance or safety applications, representing an incremental improvement in multi-modal fusion methods.
The paper tackles the challenge of multi-modal crowd counting by introducing an auxiliary broker modality to bridge the gap between visual and thermal/depth images, achieving promising results with only 4 million additional parameters.
Multi-modal crowd counting involves estimating crowd density from both visual and thermal/depth images. This task is challenging due to the significant gap between these distinct modalities. In this paper, we propose a novel approach by introducing an auxiliary broker modality and on this basis frame the task as a triple-modal learning problem. We devise a fusion-based method to generate this broker modality, leveraging a non-diffusion, lightweight counterpart of modern denoising diffusion-based fusion models. Additionally, we identify and address the ghosting effect caused by direct cross-modal image fusion in multi-modal crowd counting. Through extensive experimental evaluations on popular multi-modal crowd-counting datasets, we demonstrate the effectiveness of our method, which introduces only 4 million additional parameters, yet achieves promising results. The code is available at https://github.com/HenryCilence/Broker-Modality-Crowd-Counting.