Sideways: Depth-Parallel Training of Video Models
This addresses training inefficiency for video model developers, offering a novel approach but is incremental as it builds on existing backpropagation frameworks.
The paper tackles the problem of inefficient training of video models due to temporal synchronization in backpropagation by proposing Sideways, an approximate scheme that overwrites activations with new frames, which converges and shows potential for better generalization compared to standard methods.
We propose Sideways, an approximate backpropagation scheme for training video models. In standard backpropagation, the gradients and activations at every computation step through the model are temporally synchronized. The forward activations need to be stored until the backward pass is executed, preventing inter-layer (depth) parallelization. However, can we leverage smooth, redundant input streams such as videos to develop a more efficient training scheme? Here, we explore an alternative to backpropagation; we overwrite network activations whenever new ones, i.e., from new frames, become available. Such a more gradual accumulation of information from both passes breaks the precise correspondence between gradients and activations, leading to theoretically more noisy weight updates. Counter-intuitively, we show that Sideways training of deep convolutional video networks not only still converges, but can also potentially exhibit better generalization compared to standard synchronized backpropagation.