Learngene: From Open-World to Your Learning Task
This work addresses challenges in open-world recognition and few-shot learning for AI systems, offering a novel approach but with incremental improvements over existing methods.
The paper tackles the problem of deep learning models struggling with unknown classes in open-world scenarios and overfitting on small datasets by proposing a collective-individual paradigm inspired by biological inheritance. It introduces 'learngene' to transfer meta-knowledge from a collective model to lightweight individual models, achieving effective performance with few samples on target tasks.
Although deep learning has made significant progress on fixed large-scale datasets, it typically encounters challenges regarding improperly detecting unknown/unseen classes in the open-world scenario, over-parametrized, and overfitting small samples. Since biological systems can overcome the above difficulties very well, individuals inherit an innate gene from collective creatures that have evolved over hundreds of millions of years and then learn new skills through few examples. Inspired by this, we propose a practical collective-individual paradigm where an evolution (expandable) network is trained on sequential tasks and then recognize unknown classes in real-world. Moreover, the learngene, i.e., the gene for learning initialization rules of the target model, is proposed to inherit the meta-knowledge from the collective model and reconstruct a lightweight individual model on the target task. Particularly, a novel criterion is proposed to discover learngene in the collective model, according to the gradient information. Finally, the individual model is trained only with few samples on the target learning tasks. We demonstrate the effectiveness of our approach in an extensive empirical study and theoretical analysis.