Activation Learning by Local Competitions
This work addresses the need for alternative learning methods in machine learning, offering a novel paradigm that could unify various learning types and improve robustness, though it appears incremental as it builds upon existing biological inspirations.
The paper tackles the limitations of backpropagation by introducing activation learning, a biologically plausible local learning rule that discovers unsupervised features through neuron competitions, achieving comparable performance to backpropagation on small datasets and excelling in scenarios with fewer samples or disturbances, while also enabling tasks like image generation and anomaly detection with state-of-the-art results.
Despite its great success, backpropagation has certain limitations that necessitate the investigation of new learning methods. In this study, we present a biologically plausible local learning rule that improves upon Hebb's well-known proposal and discovers unsupervised features by local competitions among neurons. This simple learning rule enables the creation of a forward learning paradigm called activation learning, in which the output activation (sum of the squared output) of the neural network estimates the likelihood of the input patterns, or "learn more, activate more" in simpler terms. For classification on a few small classical datasets, activation learning performs comparably to backpropagation using a fully connected network, and outperforms backpropagation when there are fewer training samples or unpredictable disturbances. Additionally, the same trained network can be used for a variety of tasks, including image generation and completion. Activation learning also achieves state-of-the-art performance on several real-world datasets for anomaly detection. This new learning paradigm, which has the potential to unify supervised, unsupervised, and semi-supervised learning and is reasonably more resistant to adversarial attacks, deserves in-depth investigation.