Drop-DTW: Aligning Common Signal Between Sequences While Dropping Outliers
This addresses the challenge of robust sequence alignment for tasks like video and audio analysis, offering a solution for handling noisy data, though it is an incremental improvement over existing DTW methods.
The paper tackles the problem of sequence-to-sequence alignment in the presence of outliers by introducing Drop-DTW, a novel algorithm that aligns common signals while automatically dropping outliers, achieving state-of-the-art results in applications like temporal step localization and cross-modal retrieval under weakly- or unsupervised settings.
In this work, we consider the problem of sequence-to-sequence alignment for signals containing outliers. Assuming the absence of outliers, the standard Dynamic Time Warping (DTW) algorithm efficiently computes the optimal alignment between two (generally) variable-length sequences. While DTW is robust to temporal shifts and dilations of the signal, it fails to align sequences in a meaningful way in the presence of outliers that can be arbitrarily interspersed in the sequences. To address this problem, we introduce Drop-DTW, a novel algorithm that aligns the common signal between the sequences while automatically dropping the outlier elements from the matching. The entire procedure is implemented as a single dynamic program that is efficient and fully differentiable. In our experiments, we show that Drop-DTW is a robust similarity measure for sequence retrieval and demonstrate its effectiveness as a training loss on diverse applications. With Drop-DTW, we address temporal step localization on instructional videos, representation learning from noisy videos, and cross-modal representation learning for audio-visual retrieval and localization. In all applications, we take a weakly- or unsupervised approach and demonstrate state-of-the-art results under these settings.