Generalised Perceptron Learning
This work provides a new theoretical framework for perceptron learning, potentially simplifying the development of new algorithms for researchers working with neural networks.
This paper generalizes the perceptron learning algorithm to proximal activation functions, interpreting it as an incremental gradient method for a new energy function based on a generalized Bregman distance. This approach eliminates the need to differentiate the activation function and enables new algorithms, such as a novel iterative soft-thresholding variant for sparse perceptrons.
We present a generalisation of Rosenblatt's traditional perceptron learning algorithm to the class of proximal activation functions and demonstrate how this generalisation can be interpreted as an incremental gradient method applied to a novel energy function. This novel energy function is based on a generalised Bregman distance, for which the gradient with respect to the weights and biases does not require the differentiation of the activation function. The interpretation as an energy minimisation algorithm paves the way for many new algorithms, of which we explore a novel variant of the iterative soft-thresholding algorithm for the learning of sparse perceptrons.