Neural Architecture Search of Deep Priors: Towards Continual Learning without Catastrophic Interference
This addresses catastrophic interference in continual learning for AI systems that need to learn sequentially, though it's incremental as it builds on existing neural architecture search and deep prior concepts.
The paper tackles the problem of catastrophic interference in continual learning by demonstrating that neural architecture search can identify random-weight architectures (deep priors) that enable linear classifiers to match fully trained deep networks. The result shows these deep priors allow discrimination of new classes through simple linear training without catastrophic forgetting.
In this paper we analyze the classification performance of neural network structures without parametric inference. Making use of neural architecture search, we empirically demonstrate that it is possible to find random weight architectures, a deep prior, that enables a linear classification to perform on par with fully trained deep counterparts. Through ablation experiments, we exclude the possibility of winning a weight initialization lottery and confirm that suitable deep priors do not require additional inference. In an extension to continual learning, we investigate the possibility of catastrophic interference free incremental learning. Under the assumption of classes originating from the same data distribution, a deep prior found on only a subset of classes is shown to allow discrimination of further classes through training of a simple linear classifier.