Neuronal diversity can improve machine learning for physics and beyond
This work addresses the limitation of uniform neurons in neural networks for researchers in machine learning and physics, offering a novel approach to enhance model performance through learned diversity.
The paper tackled the problem of homogeneous neurons in artificial neural networks by constructing networks where neurons learn their own activation functions, leading to diversification and improved performance on tasks like image classification and nonlinear regression, with examples including digit classification and forecasting dynamical systems.
Diversity conveys advantages in nature, yet homogeneous neurons typically comprise the layers of artificial neural networks. Here we construct neural networks from neurons that learn their own activation functions, quickly diversify, and subsequently outperform their homogeneous counterparts on image classification and nonlinear regression tasks. Sub-networks instantiate the neurons, which meta-learn especially efficient sets of nonlinear responses. Examples include conventional neural networks classifying digits and forecasting a van der Pol oscillator and physics-informed Hamiltonian neural networks learning Hénon-Heiles stellar orbits and the swing of a video recorded pendulum clock. Such \textit{learned diversity} provides examples of dynamical systems selecting diversity over uniformity and elucidates the role of diversity in natural and artificial systems.