Active Learning for Single Neuron Models with Lipschitz Non-Linearities
This work addresses the challenge of efficiently learning single neuron models, which are important for modeling physical phenomena and PDEs, by extending a known active learning method to non-linear cases, though it is incremental as it adapts an existing linear strategy.
The paper tackles the problem of active learning for single neuron models with Lipschitz non-linearities in an agnostic setting with adversarial label noise, showing that leverage score sampling, a strategy previously used for linear functions, provides strong provable approximation guarantees and empirically outperforms uniform sampling in fitting these models.
We consider the problem of active learning for single neuron models, also sometimes called ``ridge functions'', in the agnostic setting (under adversarial label noise). Such models have been shown to be broadly effective in modeling physical phenomena, and for constructing surrogate data-driven models for partial differential equations. Surprisingly, we show that for a single neuron model with any Lipschitz non-linearity (such as the ReLU, sigmoid, absolute value, low-degree polynomial, among others), strong provable approximation guarantees can be obtained using a well-known active learning strategy for fitting \emph{linear functions} in the agnostic setting. % -- i.e. for the case when there is no non-linearity. Namely, we can collect samples via statistical \emph{leverage score sampling}, which has been shown to be near-optimal in other active learning scenarios. We support our theoretical results with empirical simulations showing that our proposed active learning strategy based on leverage score sampling outperforms (ordinary) uniform sampling when fitting single neuron models.