Zero Initialization of modified Gated Recurrent Encoder-Decoder Network for Short Term Load Forecasting
This is an incremental improvement for researchers and practitioners in machine learning, specifically in neural network initialization for time-series forecasting tasks.
The authors tackled the problem of slow learning and suboptimal accuracy in neural networks by proposing Zero Initialization (ZI) for weights in a single-layer network, which reduced epochs and improved accuracy to 0.94% error in short-term load forecasting.
Single layer Feedforward Neural Network(FNN) is used many a time as a last layer in models such as seq2seq or could be a simple RNN network. The importance of such layer is to transform the output to our required dimensions. When it comes to weights and biases initialization, there is no such specific technique that could speed up the learning process. We could depend on deep network initialization techniques such as Xavier or He initialization. But such initialization fails to show much improvement in learning speed or accuracy. In this paper we propose Zero Initialization (ZI) for weights of a single layer network. We first test this technique with on a simple RNN network and compare the results against Xavier, He and Identity initialization. As a final test we implement it on a seq2seq network. It was found that ZI considerably reduces the number of epochs used and improve the accuracy. The developed model has been applied for short-term load forecasting using the load data of Australian Energy Market. The model is able to forecast the day ahead load accurately with error of 0.94%.