Mutual Information Decay Curves and Hyper-Parameter Grid Search Design for Recurrent Neural Architectures
This work provides a systematic way to optimize hyper-parameters for recurrent neural networks, which can benefit researchers and practitioners working with sequential data by improving model performance.
This paper proposes a method to design hyper-parameter grid searches for recurrent neural networks by analyzing long-distance dependencies in datasets using mutual information. This approach led to state-of-the-art results for DilatedRNNs on several benchmark datasets.
We present an approach to design the grid searches for hyper-parameter optimization for recurrent neural architectures. The basis for this approach is the use of mutual information to analyze long distance dependencies (LDDs) within a dataset. We also report a set of experiments that demonstrate how using this approach, we obtain state-of-the-art results for DilatedRNNs across a range of benchmark datasets.