Score Function Features for Discriminative Learning: Matrix and Tensor Framework
This work addresses feature learning challenges in domains like speech, computer vision, and natural language processing by providing a framework to use generative models for discriminative tasks, though it appears incremental as it builds on existing concepts of score functions and spectral methods.
The paper tackles the problem of discriminative learning by introducing a novel class of matrix and tensor-valued features based on higher-order score functions, which capture local variations in input probability density and can be pre-trained using unlabeled samples, with efficient spectral decomposition algorithms enabling the extraction of richer discriminative information in overcomplete representations.
Feature learning forms the cornerstone for tackling challenging learning problems in domains such as speech, computer vision and natural language processing. In this paper, we consider a novel class of matrix and tensor-valued features, which can be pre-trained using unlabeled samples. We present efficient algorithms for extracting discriminative information, given these pre-trained features and labeled samples for any related task. Our class of features are based on higher-order score functions, which capture local variations in the probability density function of the input. We establish a theoretical framework to characterize the nature of discriminative information that can be extracted from score-function features, when used in conjunction with labeled samples. We employ efficient spectral decomposition algorithms (on matrices and tensors) for extracting discriminative components. The advantage of employing tensor-valued features is that we can extract richer discriminative information in the form of an overcomplete representations. Thus, we present a novel framework for employing generative models of the input for discriminative learning.