A Proposal for Networks Capable of Continual Learning
This addresses the challenge of continual learning for AI systems, though it appears incremental with increased computational complexity and representational limitations.
The paper tackles the problem of neural networks lacking the ability to retain past responses after parameter updates, which is crucial for continual learning, and proposes Modelleyen as an alternative approach that achieves continual learning without sample replay or task boundaries, as demonstrated on MNIST and a simple environment.
We analyze the ability of computational units to retain past responses after parameter updates, a key property for system-wide continual learning. Neural networks trained with gradient descent lack this capability, prompting us to propose Modelleyen, an alternative approach with inherent response preservation. We demonstrate through experiments on modeling the dynamics of a simple environment and on MNIST that, despite increased computational complexity and some representational limitations at its current stage, Modelleyen achieves continual learning without relying on sample replay or predefined task boundaries.