Complex Momentum for Optimization in Games
This addresses optimization challenges in adversarial settings like GANs, offering a drop-in replacement for standard optimizers with potential benefits for machine learning practitioners, though it appears incremental as it builds on existing momentum methods.
The paper tackles optimization in differentiable games by generalizing gradient descent with momentum to complex-valued momentum, proving convergence on bilinear zero-sum games and demonstrating improved convergence in adversarial games like GANs with minimal computational overhead, achieving better inception scores on CIFAR-10 with BigGAN.
We generalize gradient descent with momentum for optimization in differentiable games to have complex-valued momentum. We give theoretical motivation for our method by proving convergence on bilinear zero-sum games for simultaneous and alternating updates. Our method gives real-valued parameter updates, making it a drop-in replacement for standard optimizers. We empirically demonstrate that complex-valued momentum can improve convergence in realistic adversarial games - like generative adversarial networks - by showing we can find better solutions with an almost identical computational cost. We also show a practical generalization to a complex-valued Adam variant, which we use to train BigGAN to better inception scores on CIFAR-10.