Efficient Parametric Approximations of Neural Network Function Space Distance
This work addresses the need for compact model and data summaries in machine learning, offering an incremental improvement for applications like continual learning and data quality assessment.
The paper tackles the problem of efficiently estimating the Function Space Distance (FSD) between neural networks without storing full datasets, proposing the LAFTR method that approximates ReLU networks as linear networks with stochastic gating. The result is a parametric approximation that outperforms others in memory efficiency, is competitive with nonparametric state-of-the-art in continual learning, and effectively estimates influence functions and detects mislabeled examples.
It is often useful to compactly summarize important properties of model parameters and training data so that they can be used later without storing and/or iterating over the entire dataset. As a specific case, we consider estimating the Function Space Distance (FSD) over a training set, i.e. the average discrepancy between the outputs of two neural networks. We propose a Linearized Activation Function TRick (LAFTR) and derive an efficient approximation to FSD for ReLU neural networks. The key idea is to approximate the architecture as a linear network with stochastic gating. Despite requiring only one parameter per unit of the network, our approach outcompetes other parametric approximations with larger memory requirements. Applied to continual learning, our parametric approximation is competitive with state-of-the-art nonparametric approximations, which require storing many training examples. Furthermore, we show its efficacy in estimating influence functions accurately and detecting mislabeled examples without expensive iterations over the entire dataset.