Growing a Neural Network in Breadth, Depth, and Time
For neuroscientists and AI researchers, this provides a normative framework linking resource constraints to neural architecture design, offering insights into brain structure and efficient AI.
The paper introduces differentiable cost terms for breadth, depth, and time in a recurrent convolutional neural network, enabling joint optimization with task errors. Training under varying resource pressures leads to emergent computational graphs that trade off these resources, with time usage correlating with human reaction times in object recognition.
Spatial and temporal resource constraints are critical for both biological and artificial intelligent systems. Here we define differentiable cost terms for breadth, depth, and time within a recurrent convolutional neural network conceived as a finite subset of an infinite lattice. We optimize these costs jointly with task errors via backpropagation. We set different pressures on breadth, depth, and time, which leads to diverse computational graphs emerging organically through training. We find that all three resources can be traded off against each other to achieve a given level of accuracy. Networks grow in all three dimensions with task complexity and spontaneously take more recurrent steps when inputs are occluded. Surprisingly, time used by the model correlates with human reaction times in an object recognition task. Our framework provides a normative account of how resource constraints shape neural architectures, connecting to questions about brain design in neuroscience, and may help illuminate the diversity of neural solutions found in nature.