A Telescopic Binary Learning Machine for Training Neural Networks
This work addresses training efficiency and generalization for neural networks, but it is incremental as it builds on existing local search methods with adaptive bit representation.
The paper tackles neural network training by proposing a telescopic multi-scale local search algorithm with adaptive binary representation, which leads to faster search and better quality local minima, as demonstrated on benchmark tasks including a highly non-linear artificial problem, a control problem, and real-world applications.
This paper proposes a new algorithm based on multi-scale stochastic local search with binary representation for training neural networks. In particular, we study the effects of neighborhood evaluation strategies, the effect of the number of bits per weight and that of the maximum weight range used for mapping binary strings to real values. Following this preliminary investigation, we propose a telescopic multi-scale version of local search where the number of bits is increased in an adaptive manner, leading to a faster search and to local minima of better quality. An analysis related to adapting the number of bits in a dynamic way is also presented. The control on the number of bits, which happens in a natural manner in the proposed method, is effective to increase the generalization performance. Benchmark tasks include a highly non-linear artificial problem, a control problem requiring either feed-forward or recurrent architectures for feedback control, and challenging real-world tasks in different application domains. The results demonstrate the effectiveness of the proposed method.