AlignNet: A Unifying Approach to Audio-Visual Alignment
This addresses audio-visual synchronization challenges for applications like video editing and multimedia analysis, though it appears incremental as it builds on established principles.
The paper tackles the problem of synchronizing videos with reference audios under non-uniform misalignments, achieving state-of-the-art performance on dance-music and speech-lip alignment tasks.
We present AlignNet, a model that synchronizes videos with reference audios under non-uniform and irregular misalignments. AlignNet learns the end-to-end dense correspondence between each frame of a video and an audio. Our method is designed according to simple and well-established principles: attention, pyramidal processing, warping, and affinity function. Together with the model, we release a dancing dataset Dance50 for training and evaluation. Qualitative, quantitative and subjective evaluation results on dance-music alignment and speech-lip alignment demonstrate that our method far outperforms the state-of-the-art methods. Project video and code are available at https://jianrenw.github.io/AlignNet.