Large-Scale Gradient-Free Deep Learning with Recursive Local Representation Alignment
This provides a neuro-biologically-plausible alternative to backpropagation that could make large-scale deep learning more accessible by reducing hardware costs and enabling parallelization, though it is incremental as it builds on existing gradient-free methods.
The paper tackles the problem of training deep neural networks without backpropagation by proposing a gradient-free learning procedure called recursive local representation alignment, which achieves comparable generalization to backprop on CIFAR-10 and ImageNet while converging sooner due to parallelizable and less demanding weight updates.
Training deep neural networks on large-scale datasets requires significant hardware resources whose costs (even on cloud platforms) put them out of reach of smaller organizations, groups, and individuals. Backpropagation, the workhorse for training these networks, is an inherently sequential process that is difficult to parallelize. Furthermore, it requires researchers to continually develop various tricks, such as specialized weight initializations and activation functions, in order to ensure a stable parameter optimization. Our goal is to seek an effective, neuro-biologically-plausible alternative to backprop that can be used to train deep networks. In this paper, we propose a gradient-free learning procedure, recursive local representation alignment, for training large-scale neural architectures. Experiments with residual networks on CIFAR-10 and the large benchmark, ImageNet, show that our algorithm generalizes as well as backprop while converging sooner due to weight updates that are parallelizable and computationally less demanding. This is empirical evidence that a backprop-free algorithm can scale up to larger datasets.