BERT for Large-scale Video Segment Classification with Test-time Augmentation
This work addresses video understanding for applications like content tagging, but it is incremental as it builds on existing models with test-time augmentation.
The paper tackled the problem of localizing video-level labels to precise time segments in large-scale video data, achieving a MAP@100K score of 0.7871 and ranking 9th out of 283 teams in a competition.
This paper presents our approach to the third YouTube-8M video understanding competition that challenges par-ticipants to localize video-level labels at scale to the pre-cise time in the video where the label actually occurs. Ourmodel is an ensemble of frame-level models such as GatedNetVLAD and NeXtVLAD and various BERT models withtest-time augmentation. We explore multiple ways to ag-gregate BERT outputs as video representation and variousways to combine visual and audio information. We proposetest-time augmentation as shifting video frames to one leftor right unit, which adds variety to the predictions and em-pirically shows improvement in evaluation metrics. We firstpre-train the model on the 4M training video-level data, andthen fine-tune the model on 237K annotated video segment-level data. We achieve MAP@100K 0.7871 on private test-ing video segment data, which is ranked 9th over 283 teams.