Mitigating Semantic Leakage in Cross-lingual Embeddings via Orthogonality Constraint
This addresses a bottleneck in cross-lingual NLP for applications like parallel data mining, offering an incremental improvement over existing disentanglement methods.
The paper tackled the problem of semantic leakage in cross-lingual sentence embeddings, where language-specific information contaminates semantic representations, and proposed ORACLE, a training objective that enforces orthogonality to reduce leakage and improve alignment in retrieval and similarity tasks.
Accurately aligning contextual representations in cross-lingual sentence embeddings is key for effective parallel data mining. A common strategy for achieving this alignment involves disentangling semantics and language in sentence embeddings derived from multilingual pre-trained models. However, we discover that current disentangled representation learning methods suffer from semantic leakage - a term we introduce to describe when a substantial amount of language-specific information is unintentionally leaked into semantic representations. This hinders the effective disentanglement of semantic and language representations, making it difficult to retrieve embeddings that distinctively represent the meaning of the sentence. To address this challenge, we propose a novel training objective, ORthogonAlity Constraint LEarning (ORACLE), tailored to enforce orthogonality between semantic and language embeddings. ORACLE builds upon two components: intra-class clustering and inter-class separation. Through experiments on cross-lingual retrieval and semantic textual similarity tasks, we demonstrate that training with the ORACLE objective effectively reduces semantic leakage and enhances semantic alignment within the embedding space.