Improving Optimization for Models With Continuous Symmetry Breaking
This addresses a problem for researchers and practitioners in representation learning by improving optimization efficiency, though it is incremental as it builds on existing symmetry-breaking concepts.
The paper tackled slow convergence in representation learning models with continuous symmetry by proposing a new optimization algorithm based on gauge theory, achieving orders of magnitude faster convergence and more interpretable representations in dynamic matrix factorization and word embedding models.
Many loss functions in representation learning are invariant under a continuous symmetry transformation. For example, the loss function of word embeddings (Mikolov et al., 2013) remains unchanged if we simultaneously rotate all word and context embedding vectors. We show that representation learning models for time series possess an approximate continuous symmetry that leads to slow convergence of gradient descent. We propose a new optimization algorithm that speeds up convergence using ideas from gauge theory in physics. Our algorithm leads to orders of magnitude faster convergence and to more interpretable representations, as we show for dynamic extensions of matrix factorization and word embedding models. We further present an example application of our proposed algorithm that translates modern words into their historic equivalents.