Video Activity Localisation with Uncertainties in Temporal Boundary
This solves the issue of ambiguous activity boundaries in unscripted videos for video-text correlation tasks, representing an incremental improvement with a novel method for a known bottleneck.
The paper tackles the problem of video activity localisation by addressing uncertainties in temporal boundary annotations, which are often ignored in training and lead to poor generalisation. It introduces Elastic Moment Bounding (EMB) to adaptively model flexible boundaries, achieving compelling advantages over existing methods on three benchmarks.
Current methods for video activity localisation over time assume implicitly that activity temporal boundaries labelled for model training are determined and precise. However, in unscripted natural videos, different activities mostly transit smoothly, so that it is intrinsically ambiguous to determine in labelling precisely when an activity starts and ends over time. Such uncertainties in temporal labelling are currently ignored in model training, resulting in learning mis-matched video-text correlation with poor generalisation in test. In this work, we solve this problem by introducing Elastic Moment Bounding (EMB) to accommodate flexible and adaptive activity temporal boundaries towards modelling universally interpretable video-text correlation with tolerance to underlying temporal uncertainties in pre-fixed annotations. Specifically, we construct elastic boundaries adaptively by mining and discovering frame-wise temporal endpoints that can maximise the alignment between video segments and query sentences. To enable both more accurate matching (segment content attention) and more robust localisation (segment elastic boundaries), we optimise the selection of frame-wise endpoints subject to segment-wise contents by a novel Guided Attention mechanism. Extensive experiments on three video activity localisation benchmarks demonstrate compellingly the EMB's advantages over existing methods without modelling uncertainty.