Audio Concept Classification with Hierarchical Deep Neural Networks
This work addresses the problem of improving audio-based multimedia retrieval for applications like video indexing, though it is incremental as it adapts deep learning to a specific domain.
The paper tackles audio concept classification in user-generated videos by introducing a hierarchical deep neural network that analyzes short- and long-term context, outperforming Gaussian-Mixture-Models by 54%, a Neural Network by 33%, and a Deep Neural Network by 12% on the TRECVID-MED database.
Audio-based multimedia retrieval tasks may identify semantic information in audio streams, i.e., audio concepts (such as music, laughter, or a revving engine). Conventional Gaussian-Mixture-Models have had some success in classifying a reduced set of audio concepts. However, multi-class classification can benefit from context window analysis and the discriminating power of deeper architectures. Although deep learning has shown promise in various applications such as speech and object recognition, it has not yet met the expectations for other fields such as audio concept classification. This paper explores, for the first time, the potential of deep learning in classifying audio concepts on User-Generated Content videos. The proposed system is comprised of two cascaded neural networks in a hierarchical configuration to analyze the short- and long-term context information. Our system outperforms a GMM approach by a relative 54%, a Neural Network by 33%, and a Deep Neural Network by 12% on the TRECVID-MED database