Approximating Activation Functions
This work addresses the problem of slow training times for neural networks using complex activation functions, offering practical speed-ups for machine learning practitioners, though it is incremental as it builds on existing approximation techniques.
The paper tackled the computational expense of hyperbolic tangent and sigmoid activation functions in neural networks, particularly in recurrent networks like LSTMs, by developing function approximations that achieved 10% to 53% improvements in training times on the CPU.
ReLU is widely seen as the default choice for activation functions in neural networks. However, there are cases where more complicated functions are required. In particular, recurrent neural networks (such as LSTMs) make extensive use of both hyperbolic tangent and sigmoid functions. These functions are expensive to compute. We used function approximation techniques to develop replacements for these functions and evaluated them empirically on three popular network configurations. We find safe approximations that yield a 10% to 37% improvement in training times on the CPU. These approximations were suitable for all cases we considered and we believe are appropriate replacements for all networks using these activation functions. We also develop ranged approximations which only apply in some cases due to restrictions on their input domain. Our ranged approximations yield a performance improvement of 20% to 53% in network training time. Our functions also match or considerably out perform the ad-hoc approximations used in Theano and the implementation of Word2Vec.