BERT-JEPA: Reorganizing CLS Embeddings for Language-Invariant Semantics
This addresses the issue of language invariance in multilingual NLP models, though it appears incremental as it builds on existing BERT and JEPA techniques.
The paper tackled the problem of collapsed [CLS] embedding spaces in BERT-style models by introducing BERT-JEPA, which adds a JEPA training objective to reorganize embeddings into a language-agnostic space, resulting in increased performance across multilingual benchmarks.
Joint Embedding Predictive Architectures (JEPA) are a novel self supervised training technique that have shown recent promise across domains. We introduce BERT-JEPA (BEPA), a training paradigm that adds a JEPA training objective to BERT-style models, working to combat a collapsed [CLS] embedding space and turning it into a language-agnostic space. This new structure leads to increased performance across multilingual benchmarks.