Cultural Event Recognition with Visual ConvNets and Temporal Models
This work addresses cultural event recognition for computer vision applications, but it is incremental as it builds on existing methods like CNNs and SVMs with a temporal refinement step.
The paper tackled the problem of automatically classifying images from 50 cultural events by combining visual features from convolutional neural networks with temporal information using a hierarchical classifier scheme, achieving a mean average precision of 0.767 and the second-best result in the ChaLearn Challenge 2015.
This paper presents our contribution to the ChaLearn Challenge 2015 on Cultural Event Classification. The challenge in this task is to automatically classify images from 50 different cultural events. Our solution is based on the combination of visual features extracted from convolutional neural networks with temporal information using a hierarchical classifier scheme. We extract visual features from the last three fully connected layers of both CaffeNet (pretrained with ImageNet) and our fine tuned version for the ChaLearn challenge. We propose a late fusion strategy that trains a separate low-level SVM on each of the extracted neural codes. The class predictions of the low-level SVMs form the input to a higher level SVM, which gives the final event scores. We achieve our best result by adding a temporal refinement step into our classification scheme, which is applied directly to the output of each low-level SVM. Our approach penalizes high classification scores based on visual features when their time stamp does not match well an event-specific temporal distribution learned from the training and validation data. Our system achieved the second best result in the ChaLearn Challenge 2015 on Cultural Event Classification with a mean average precision of 0.767 on the test set.