Training Across Reservoirs: Using Numerical Differentiation To Couple Trainable Networks With Black-Box Reservoirs
This addresses a technical bottleneck for researchers developing hybrid trainable architectures that combine digital and analogue components, but it appears incremental as an improved perturbative method.
The paper tackles the problem of training neural networks that integrate black-box functions with inaccessible computational graphs by introducing Bounded Numerical Differentiation (BOND), a perturbative method for estimating partial derivatives. The result shows that using fixed, untrained networks as black-box functions can enhance model performance without increasing trainable parameters, though no concrete numerical improvements are provided.
We introduce Bounded Numerical Differentiation (BOND), a perturbative method for estimating partial derivatives across network structures with inaccessible computational graphs. BOND demonstrates improved accuracy and scalability from existing perturbative methods, enabling new explorations of trainable architectures that integrate black-box functions. We observe that these black-box functions, realized in our experiments as fixed, untrained networks, can enhance model performance without increasing the number of trainable parameters. This improvement is achieved without extensive optimization of the architecture or properties of the black-box function itself. Our findings highlight the potential of leveraging fixed, non-trainable modules to expand model capacity, suggesting a path toward combining analogue and digital devices as a mechanism for scaling networks.