Unsupervised Alignment of Embeddings with Wasserstein Procrustes
This addresses the problem of unsupervised alignment for tasks like bilingual lexicon induction, offering an efficient alternative to adversarial methods, though it is incremental as it builds on existing alignment techniques.
The paper tackles the problem of aligning two sets of high-dimensional points, such as word embeddings for unsupervised translation, by proposing a method based on joint estimation of orthogonal and permutation matrices with a convex relaxation initialization. It achieves state-of-the-art results on unsupervised word translation with reduced computational resources.
We consider the task of aligning two sets of points in high dimension, which has many applications in natural language processing and computer vision. As an example, it was recently shown that it is possible to infer a bilingual lexicon, without supervised data, by aligning word embeddings trained on monolingual data. These recent advances are based on adversarial training to learn the mapping between the two embeddings. In this paper, we propose to use an alternative formulation, based on the joint estimation of an orthogonal matrix and a permutation matrix. While this problem is not convex, we propose to initialize our optimization algorithm by using a convex relaxation, traditionally considered for the graph isomorphism problem. We propose a stochastic algorithm to minimize our cost function on large scale problems. Finally, we evaluate our method on the problem of unsupervised word translation, by aligning word embeddings trained on monolingual data. On this task, our method obtains state of the art results, while requiring less computational resources than competing approaches.