Neural Feature Learning in Function Space
This work addresses foundational challenges in machine learning by providing a unified theoretical framework for feature learning, which is incremental in extending classical approaches to neural settings.
The paper tackles the problem of designing learning systems by introducing a feature geometry framework that unifies statistical dependence and feature representations in function space, resulting in systematic algorithm designs for optimal feature learning using standard neural networks and optimizers.
We present a novel framework for learning system design with neural feature extractors. First, we introduce the feature geometry, which unifies statistical dependence and feature representations in a function space equipped with inner products. This connection defines function-space concepts on statistical dependence, such as norms, orthogonal projection, and spectral decomposition, exhibiting clear operational meanings. In particular, we associate each learning setting with a dependence component and formulate learning tasks as finding corresponding feature approximations. We propose a nesting technique, which provides systematic algorithm designs for learning the optimal features from data samples with off-the-shelf network architectures and optimizers. We further demonstrate multivariate learning applications, including conditional inference and multimodal learning, where we present the optimal features and reveal their connections to classical approaches.