Emergent Structures and Lifetime Structure Evolution in Artificial Neural Networks
This work addresses the need for more flexible and adaptive neural network architectures for machine learning practitioners, though it appears incremental as it builds on existing gradient descent methods to enable structural evolution.
The paper tackled the problem of enabling artificial neural networks to dynamically adapt their connectivity structures during training, similar to biological neural networks, by introducing the Unstructured Recursive Network (URN). It demonstrated that various common network structures, such as fully connected and residual networks, can emerge from the same URN through gradient descent on a general loss function influenced by data structure and regulators.
Motivated by the flexibility of biological neural networks whose connectivity structure changes significantly during their lifetime, we introduce the Unstructured Recursive Network (URN) and demonstrate that it can exhibit similar flexibility during training via gradient descent. We show empirically that many of the different neural network structures commonly used in practice today (including fully connected, locally connected and residual networks of different depths and widths) can emerge dynamically from the same URN. These different structures can be derived using gradient descent on a single general loss function where the structure of the data and the relative strengths of various regulator terms determine the structure of the emergent network. We show that this loss function and the regulators arise naturally when considering the symmetries of the network as well as the geometric properties of the input data.