Deepening Neural Networks Implicitly and Locally via Recurrent Attention Strategy
This work addresses efficiency challenges in deep learning for practitioners, though it is incremental as it builds on existing attention mechanisms.
The authors tackled the problem of increasing neural network depth without significantly raising computational cost and parameter size by proposing a Recurrent Attention Strategy (RAS), which improved performance on three benchmark datasets with minimal overhead.
More and more empirical and theoretical evidence shows that deepening neural networks can effectively improve their performance under suitable training settings. However, deepening the backbone of neural networks will inevitably and significantly increase computation and parameter size. To mitigate these problems, we propose a simple-yet-effective Recurrent Attention Strategy (RAS), which implicitly increases the depth of neural networks with lightweight attention modules by local parameter sharing. The extensive experiments on three widely-used benchmark datasets demonstrate that RAS can improve the performance of neural networks at a slight addition of parameter size and computation, performing favorably against other existing well-known attention modules.