Convergence and scaling of Boolean-weight optimization for hardware reservoirs
This work provides a foundation for efficient optimization in hardware neural networks, addressing a bottleneck for implementing scalable AI solutions, though it is incremental as it builds on existing methods.
The paper tackled the problem of optimizing large parameter spaces in hardware neural networks by analytically deriving scaling laws for Coordinate Descent applied to reservoir readout layers, showing exponential convergence that scales linearly with neuron count and matching experimental results from a photonic reservoir.
Hardware implementation of neural network are an essential step to implement next generation efficient and powerful artificial intelligence solutions. Besides the realization of a parallel, efficient and scalable hardware architecture, the optimization of the system's extremely large parameter space with sampling-efficient approaches is essential. Here, we analytically derive the scaling laws for highly efficient Coordinate Descent applied to optimizing the readout layer of a random recurrently connection neural network, a reservoir. We demonstrate that the convergence is exponential and scales linear with the network's number of neurons. Our results perfectly reproduce the convergence and scaling of a large-scale photonic reservoir implemented in a proof-of-concept experiment. Our work therefore provides a solid foundation for such optimization in hardware networks, and identifies future directions that are promising for optimizing convergence speed during learning leveraging measures of a neural network's amplitude statistics and the weight update rule.