Energy-Efficient Information Representation in MNIST Classification Using Biologically Inspired Learning
This work addresses energy and scalability issues in AI, particularly relevant for large models like LLMs, though it is incremental as it builds on prior biologically inspired methods.
The paper tackles the problem of overparameterization and energy inefficiency in artificial neural networks by proposing a biologically inspired learning rule that optimizes synaptic usage on MNIST classification, achieving higher efficiency and storage capacity than backpropagation while eliminating the need for pre-optimized architectures.
Efficient representation learning is essential for optimal information storage and classification. However, it is frequently overlooked in artificial neural networks (ANNs). This neglect results in networks that can become overparameterized by factors of up to 13, increasing redundancy and energy consumption. As the demand for large language models (LLMs) and their scale increase, these issues are further highlighted, raising significant ethical and environmental concerns. We analyze our previously developed biologically inspired learning rule using information-theoretic concepts, evaluating its efficiency on the MNIST classification task. The proposed rule, which emulates the brain's structural plasticity, naturally prevents overparameterization by optimizing synaptic usage and retaining only the essential number of synapses. Furthermore, it outperforms backpropagation (BP) in terms of efficiency and storage capacity. It also eliminates the need for pre-optimization of network architecture, enhances adaptability, and reflects the brain's ability to reserve 'space' for new memories. This approach advances scalable and energy-efficient AI and provides a promising framework for developing brain-inspired models that optimize resource allocation and adaptability.