Transformer based Multilingual document Embedding model
This work addresses the need for efficient and accurate multilingual document embeddings, but it is incremental as it builds on the existing LASER framework with transformer enhancements.
The paper tackles the problem of multilingual document embedding by proposing T-LASER, a transformer-based model that replaces BiLSTM layers with attention-based transformers for better handling of longer texts and faster parallel computations, and introduces a distance constraint loss to improve embeddings; it shows that cT-LASER significantly outperforms previous models, though specific numbers are not provided.
One of the current state-of-the-art multilingual document embedding model LASER is based on the bidirectional LSTM neural machine translation model. This paper presents a transformer-based sentence/document embedding model, T-LASER, which makes three significant improvements. Firstly, the BiLSTM layers is replaced by the attention-based transformer layers, which is more capable of learning sequential patterns in longer texts. Secondly, due to the absence of recurrence, T-LASER enables faster parallel computations in the encoder to generate the text embedding. Thirdly, we augment the NMT translation loss function with an additional novel distance constraint loss. This distance constraint loss would further bring the embeddings of parallel sentences close together in the vector space; we call the T-LASER model trained with distance constraint, cT-LASER. Our cT-LASER model significantly outperforms both BiLSTM-based LASER and the simpler transformer-based T-LASER.