Evolution Strategies Converges to Finite Differences
It clarifies a theoretical connection for researchers in optimization and reinforcement learning, but is incremental as it builds on prior investigations.
This paper tackles the relationship between Evolution Strategies (ES) and Finite Differences (FD) gradients, proving that they converge as the dimension of the optimization vector increases.
Since the debut of Evolution Strategies (ES) as a tool for Reinforcement Learning by Salimans et al. 2017, there has been interest in determining the exact relationship between the Evolution Strategies gradient and the gradient of a similar class of algorithms, Finite Differences (FD).(Zhang et al. 2017, Lehman et al. 2018) Several investigations into the subject have been performed, investigating the formal motivational differences(Lehman et al. 2018) between ES and FD, as well as the differences in a standard benchmark problem in Machine Learning, the MNIST classification problem(Zhang et al. 2017). This paper proves that while the gradients are different, they converge as the dimension of the vector under optimization increases.