Tensor-Based Backpropagation in Neural Networks with Non-Sequential Input
This work addresses a computational bottleneck for researchers and practitioners in machine learning, but it appears incremental as it builds on existing batch training methods.
The paper tackles the computational inefficiency of batch training in neural networks by proposing a tensor-based backpropagation method, which increases computational efficiency by enabling non-linear training processes.
Neural networks have been able to achieve groundbreaking accuracy at tasks conventionally considered only doable by humans. Using stochastic gradient descent, optimization in many dimensions is made possible, albeit at a relatively high computational cost. By splitting training data into batches, networks can be distributed and trained vastly more efficiently and with minimal accuracy loss. We have explored the mathematics behind efficiently implementing tensor-based batch backpropagation algorithms. A common approach to batch training is iterating over batch items individually. Explicitly using tensor operations to backpropagate allows training to be performed non-linearly, increasing computational efficiency.