Low-Cost Recurrent Neural Network Expected Performance Evaluation
This work addresses hyper-parameter optimization for recurrent neural networks, offering a more practical alternative to resource-intensive methods, though it appears incremental.
The paper tackles the problem of high computational cost in evaluating recurrent neural network hyper-parameter configurations by proposing a low-cost model based on random weight error distributions, empirically validated in three use cases to reduce exploration costs.
Recurrent neural networks are a powerful tool, but they are very sensitive to their hyper-parameter configuration. Moreover, training properly a recurrent neural network is a tough task, therefore selecting an appropriate configuration is critical. Varied strategies have been proposed to tackle this issue. However, most of them are still impractical because of the time/resources needed. In this study, we propose a low computational cost model to evaluate the expected performance of a given architecture based on the distribution of the error of random samples of the weights. We empirically validate our proposal using three use cases. The results suggest that this is a promising alternative to reduce the cost of exploration for hyper-parameter optimization.