Circuit realization and hardware linearization of monotone operator equilibrium networks
This provides a novel analog hardware implementation method for neural networks, potentially enabling more efficient and direct physical computation, though it appears incremental in extending existing circuit concepts to neural networks.
The authors demonstrated that resistor-diode networks can physically implement monotone operator equilibrium networks (a type of infinite-depth neural network), enabling analog hardware realization and direct gradient computation through hardware linearization for in-hardware training, validated via device-level circuit simulation.
It is shown that the port behavior of a resistor-diode network corresponds to the solution of a ReLU monotone operator equilibrium network (a neural network in the limit of infinite depth), giving a parsimonious construction of a neural network in analog hardware. We furthermore show that the gradient of such a circuit can be computed directly in hardware, using a procedure we call hardware linearization. This allows the network to be trained in hardware, which we demonstrate with a device-level circuit simulation. We extend the results to cascades of resistor-diode networks, which can be used to implement feedforward and other asymmetric networks. We finally show that different nonlinear elements give rise to different activation functions, and introduce the novel diode ReLU which is induced by a non-ideal diode model.