On Polynomial Approximation of Activation Function
arXiv:2202.00004v18 citations
Originality Synthesis-oriented
AI Analysis
This work addresses a domain-specific problem in neural network optimization, but it appears incremental as it builds on existing least squares methods.
The paper tackles the problem of approximating activation functions using low-degree polynomials by extending the ordinary least squares method to include gradient information in the cost function, resulting in a novel approach for function approximation.
In this work, we propose an interesting method that aims to approximate an activation function over some domain by polynomials of the presupposing low degree. The main idea behind this method can be seen as an extension of the ordinary least square method and includes the gradient of activation function into the cost function to minimize.