The Role of the Input in Natural Language Video Description
This study is significant for researchers and developers working on NLVD systems, as it highlights the impact of visual input quality and dataset integrity on model generalization, addressing a known limitation of current state-of-the-art approaches.
This paper investigates the role of visual input in Natural Language Video Description (NLVD) systems by applying data augmentation techniques to model real-world camera distortions, noise, lighting, and camera positioning. It also addresses issues with the commonly used MSVD dataset, manually correcting syntactic and semantic errors to create MSVD-v2.
Natural Language Video Description (NLVD) has recently received strong interest in the Computer Vision, Natural Language Processing (NLP), Multimedia, and Autonomous Robotics communities. The State-of-the-Art (SotA) approaches obtained remarkable results when tested on the benchmark datasets. However, those approaches poorly generalize to new datasets. In addition, none of the existing works focus on the processing of the input to the NLVD systems, which is both visual and textual. In this work, it is presented an extensive study dealing with the role of the visual input, evaluated with respect to the overall NLP performance. This is achieved performing data augmentation of the visual component, applying common transformations to model camera distortions, noise, lighting, and camera positioning, that are typical in real-world operative scenarios. A t-SNE based analysis is proposed to evaluate the effects of the considered transformations on the overall visual data distribution. For this study, it is considered the English subset of Microsoft Research Video Description (MSVD) dataset, which is used commonly for NLVD. It was observed that this dataset contains a relevant amount of syntactic and semantic errors. These errors have been amended manually, and the new version of the dataset (called MSVD-v2) is used in the experimentation. The MSVD-v2 dataset is released to help to gain insight into the NLVD problem.