Mixed Newton Method for Optimization in Complex Spaces
This is an incremental improvement for optimization in machine learning, potentially benefiting researchers and practitioners in neural network training.
The authors adapted the Mixed Newton Method from complex to real optimization by extending functions to complex space, showing that arbitrary regularizations preserve local convergence properties while preventing convergence to complex minima. They applied variants to neural network training with real and complex parameters.
In this paper, we modify and apply the recently introduced Mixed Newton Method, which is originally designed for minimizing real-valued functions of complex variables, to the minimization of real-valued functions of real variables by extending the functions to complex space. We show that arbitrary regularizations preserve the favorable local convergence properties of the method, and construct a special type of regularization used to prevent convergence to complex minima. We compare several variants of the method applied to training neural networks with real and complex parameters.