Improving Learning to Optimize Using Parameter Symmetries
This addresses meta-optimization efficiency for machine learning practitioners, though it appears incremental as it builds on prior work on symmetry transformations.
The paper tackled the problem of improving learning-to-optimize algorithms by exploiting parameter space symmetries, showing that this approach locally resembles Newton's method theoretically and can learn correct symmetry transformations during training.
We analyze a learning-to-optimize (L2O) algorithm that exploits parameter space symmetry to enhance optimization efficiency. Prior work has shown that jointly learning symmetry transformations and local updates improves meta-optimizer performance. Supporting this, our theoretical analysis demonstrates that even without identifying the optimal group element, the method locally resembles Newton's method. We further provide an example where the algorithm provably learns the correct symmetry transformation during training. To empirically evaluate L2O with teleportation, we introduce a benchmark, analyze its success and failure cases, and show that enhancements like momentum further improve performance. Our results highlight the potential of leveraging neural network parameter space symmetry to advance meta-optimization.