Learning to Adaptively Scale Recurrent Neural Networks
This work addresses the limitation of rigid scaling in RNNs for time series analysis, offering a more flexible approach for researchers and practitioners in sequence modeling, though it is incremental as it builds on existing multiscale RNN methods.
The paper tackled the problem of fixed scales in multiscale recurrent neural networks (RNNs) by proposing Adaptively Scaled RNNs (ASRNN), which learn and adjust scales based on temporal contexts, resulting in better performance on sequence modeling tasks compared to baselines without dynamical scaling.
Recent advancements in recurrent neural network (RNN) research have demonstrated the superiority of utilizing multiscale structures in learning temporal representations of time series. Currently, most of multiscale RNNs use fixed scales, which do not comply with the nature of dynamical temporal patterns among sequences. In this paper, we propose Adaptively Scaled Recurrent Neural Networks (ASRNN), a simple but efficient way to handle this problem. Instead of using predefined scales, ASRNNs are able to learn and adjust scales based on different temporal contexts, making them more flexible in modeling multiscale patterns. Compared with other multiscale RNNs, ASRNNs are bestowed upon dynamical scaling capabilities with much simpler structures, and are easy to be integrated with various RNN cells. The experiments on multiple sequence modeling tasks indicate ASRNNs can efficiently adapt scales based on different sequence contexts and yield better performances than baselines without dynamical scaling abilities.