Deep Latent Space Learning for Cross-modal Mapping of Audio and Visual Signals
This work addresses cross-modal biometric applications, offering an incremental improvement by eliminating the need for pairwise supervision while maintaining competitive results.
The paper tackles the problem of learning a shared deep latent space for audio-visual signals without pairwise supervision, achieving state-of-the-art performance on cross-modal verification and matching tasks on the VoxCeleb dataset.
We propose a novel deep training algorithm for joint representation of audio and visual information which consists of a single stream network (SSNet) coupled with a novel loss function to learn a shared deep latent space representation of multimodal information. The proposed framework characterizes the shared latent space by leveraging the class centers which helps to eliminate the need for pairwise or triplet supervision. We quantitatively and qualitatively evaluate the proposed approach on VoxCeleb, a benchmarks audio-visual dataset on a multitude of tasks including cross-modal verification, cross-modal matching, and cross-modal retrieval. State-of-the-art performance is achieved on cross-modal verification and matching while comparable results are observed on the remaining applications. Our experiments demonstrate the effectiveness of the technique for cross-modal biometric applications.