Spikes as regularizers
This work addresses overfitting in machine learning algorithms, but it appears incremental as it builds on existing methods like the perceptron and AROW with a novel regularization approach.
The paper tackled the problem of overfitting in single-layer feed-forward learning by introducing SPIRAL, a confidence-based algorithm that uses activation spikes for regularization, resulting in improved robustness and reduced overfitting compared to the averaged perceptron and AROW.
We present a confidence-based single-layer feed-forward learning algorithm SPIRAL (Spike Regularized Adaptive Learning) relying on an encoding of activation spikes. We adaptively update a weight vector relying on confidence estimates and activation offsets relative to previous activity. We regularize updates proportionally to item-level confidence and weight-specific support, loosely inspired by the observation from neurophysiology that high spike rates are sometimes accompanied by low temporal precision. Our experiments suggest that the new learning algorithm SPIRAL is more robust and less prone to overfitting than both the averaged perceptron and AROW.