Pathway-based Progressive Inference (PaPI) for Energy-Efficient Continual Learning
This addresses the challenge of enabling continual learning in resource-constrained environments, representing a novel theoretical advancement with potential domain-specific applications.
The paper tackles the problem of catastrophic forgetting and energy inefficiency in continual learning by introducing the Pathway-based Progressive Inference (PaPI) framework, achieving an O(K) improvement in the stability-plasticity trade-off and proving stronger guarantees against forgetting than Elastic Weight Consolidation while maintaining better energy efficiency.
Continual learning systems face the dual challenge of preventing catastrophic forgetting while maintaining energy efficiency, particularly in resource-constrained environments. This paper introduces Pathway-based Progressive Inference (PaPI), a novel theoretical framework that addresses these challenges through a mathematically rigorous approach to pathway selection and adaptation. We formulate continual learning as an energy-constrained optimization problem and provide formal convergence guarantees for our pathway routing mechanisms. Our theoretical analysis demonstrates that PaPI achieves an $\mathcal{O}(K)$ improvement in the stability-plasticity trade-off compared to monolithic architectures, where $K$ is the number of pathways. We derive tight bounds on forgetting rates using Fisher Information Matrix analysis and prove that PaPI's energy consumption scales with the number of active parameters rather than the total model size. Comparative theoretical analysis shows that PaPI provides stronger guarantees against catastrophic forgetting than Elastic Weight Consolidation (EWC) while maintaining better energy efficiency than both EWC and Gradient Episodic Memory (GEM). Our experimental validation confirms these theoretical advantages across multiple benchmarks, demonstrating PaPI's effectiveness for continual learning in energy-constrained settings. Our codes are available at https://github.com/zser092/PAPI_FILES.