Feed-Forward Networks with Attention Can Solve Some Long-Term Memory Problems
This addresses memory limitations in neural networks for sequence processing, though it appears incremental as it builds on existing attention mechanisms.
The authors tackled the challenge of long-term memory in sequences by proposing a simplified attention model for feed-forward networks, achieving state-of-the-art results on synthetic addition and multiplication tasks with longer and more varied sequence lengths than previously published.
We propose a simplified model of attention which is applicable to feed-forward neural networks and demonstrate that the resulting model can solve the synthetic "addition" and "multiplication" long-term memory problems for sequence lengths which are both longer and more widely varying than the best published results for these tasks.