ModEFormer: Modality-Preserving Embedding for Audio-Video Synchronization using Transformers
This addresses synchronization issues for viewers in media applications, representing an incremental improvement over existing methods.
The paper tackled the problem of audio-video synchronization in broadcasts and video conferencing by proposing ModEFormer, a method that uses modality-specific transformers to preserve input streams, achieving state-of-the-art performance of 94.5% on LRS2 and 90.9% on LRS3.
Lack of audio-video synchronization is a common problem during television broadcasts and video conferencing, leading to an unsatisfactory viewing experience. A widely accepted paradigm is to create an error detection mechanism that identifies the cases when audio is leading or lagging. We propose ModEFormer, which independently extracts audio and video embeddings using modality-specific transformers. Different from the other transformer-based approaches, ModEFormer preserves the modality of the input streams which allows us to use a larger batch size with more negative audio samples for contrastive learning. Further, we propose a trade-off between the number of negative samples and number of unique samples in a batch to significantly exceed the performance of previous methods. Experimental results show that ModEFormer achieves state-of-the-art performance, 94.5% for LRS2 and 90.9% for LRS3. Finally, we demonstrate how ModEFormer can be used for offset detection for test clips.