Continual Learning with Deep Streaming Regularized Discriminant Analysis
This work addresses the need for streaming learning in real-world applications where data arrives incrementally, offering an incremental improvement over existing methods.
The paper tackled the problem of catastrophic forgetting in continual learning by proposing a streaming version of regularized discriminant analysis, demonstrating that it outperforms batch learning and existing streaming algorithms on the ImageNet ILSVRC-2012 dataset.
Continual learning is increasingly sought after in real world machine learning applications, as it enables learning in a more human-like manner. Conventional machine learning approaches fail to achieve this, as incrementally updating the model with non-identically distributed data leads to catastrophic forgetting, where existing representations are overwritten. Although traditional continual learning methods have mostly focused on batch learning, which involves learning from large collections of labeled data sequentially, this approach is not well-suited for real-world applications where we would like new data to be integrated directly. This necessitates a paradigm shift towards streaming learning. In this paper, we propose a streaming version of regularized discriminant analysis as a solution to this challenge. We combine our algorithm with a convolutional neural network and demonstrate that it outperforms both batch learning and existing streaming learning algorithms on the ImageNet ILSVRC-2012 dataset.