U-TELL: Unsupervised Task Expert Lifelong Learning
This addresses the challenge of learning new tasks sequentially without labels for real-world ML applications, though it appears incremental as it builds on existing unsupervised CL methods.
The paper tackled the problem of catastrophic forgetting in continual learning with limited labels by proposing U-TELL, an unsupervised model that introduces task experts and uses generated structured samples, outperforming five state-of-the-art methods on seven benchmarks and an industry dataset with over 6 times faster training.
Continual learning (CL) models are designed to learn new tasks arriving sequentially without re-training the network. However, real-world ML applications have very limited label information and these models suffer from catastrophic forgetting. To address these issues, we propose an unsupervised CL model with task experts called Unsupervised Task Expert Lifelong Learning (U-TELL) to continually learn the data arriving in a sequence addressing catastrophic forgetting. During training of U-TELL, we introduce a new expert on arrival of a new task. Our proposed architecture has task experts, a structured data generator and a task assigner. Each task expert is composed of 3 blocks; i) a variational autoencoder to capture the task distribution and perform data abstraction, ii) a k-means clustering module, and iii) a structure extractor to preserve latent task data signature. During testing, task assigner selects a suitable expert to perform clustering. U-TELL does not store or replay task samples, instead, we use generated structured samples to train the task assigner. We compared U-TELL with five SOTA unsupervised CL methods. U-TELL outperformed all baselines on seven benchmarks and one industry dataset for various CL scenarios with a training time over 6 times faster than the best performing baseline.