Memory-Modular Classification: Learning to Generalize with Memory Replacement
This addresses the challenge of adapting classification models to new classes efficiently, though it appears incremental by building on memory-based and modular learning approaches.
The paper tackles the problem of enabling image classification models to generalize to new classes without retraining by separating knowledge memorization from reasoning, storing knowledge in an external memory of web-crawled data and dynamically selecting relevant content at inference time, achieving promising performance in tasks like zero-shot/few-shot classification.
We propose a novel memory-modular learner for image classification that separates knowledge memorization from reasoning. Our model enables effective generalization to new classes by simply replacing the memory contents, without the need for model retraining. Unlike traditional models that encode both world knowledge and task-specific skills into their weights during training, our model stores knowledge in the external memory of web-crawled image and text data. At inference time, the model dynamically selects relevant content from the memory based on the input image, allowing it to adapt to arbitrary classes by simply replacing the memory contents. The key differentiator that our learner meta-learns to perform classification tasks with noisy web data from unseen classes, resulting in robust performance across various classification scenarios. Experimental results demonstrate the promising performance and versatility of our approach in handling diverse classification tasks, including zero-shot/few-shot classification of unseen classes, fine-grained classification, and class-incremental classification.