Locality Preserving Loss: Neighbors that Live together, Align together
This addresses the challenge of aligning embeddings with limited data for tasks such as semantic similarity and cross-lingual alignment, though it is incremental as it builds on existing embedding methods.
The paper tackles the problem of aligning vector space embeddings from different manifolds by introducing a locality preserving loss (LPL) that maintains local neighborhoods during projection, reducing the dataset size needed for tasks like cross-lingual word alignment and showing up to 16% improvement in low-resource settings.
We present a locality preserving loss (LPL) that improves the alignment between vector space embeddings while separating uncorrelated representations. Given two pretrained embedding manifolds, LPL optimizes a model to project an embedding and maintain its local neighborhood while aligning one manifold to another. This reduces the overall size of the dataset required to align the two in tasks such as cross-lingual word alignment. We show that the LPL-based alignment between input vector spaces acts as a regularizer, leading to better and consistent accuracy than the baseline, especially when the size of the training set is small. We demonstrate the effectiveness of LPL optimized alignment on semantic text similarity (STS), natural language inference (SNLI), multi-genre language inference (MNLI) and cross-lingual word alignment(CLA) showing consistent improvements, finding up to 16% improvement over our baseline in lower resource settings.