LAWDR: Language-Agnostic Weighted Document Representations from Pre-trained Models
This work addresses the problem of efficient cross-lingual document understanding for resource-limited settings, offering an incremental improvement over existing methods.
The paper tackled the challenges of high computational costs and resource requirements for cross-lingual document representation learning by proposing an unsupervised method called LAWDR, which leverages pre-trained sentence embeddings without fine-tuning and achieved comparable performance to state-of-the-art models on benchmark datasets for cross-lingual document alignment.
Cross-lingual document representations enable language understanding in multilingual contexts and allow transfer learning from high-resource to low-resource languages at the document level. Recently large pre-trained language models such as BERT, XLM and XLM-RoBERTa have achieved great success when fine-tuned on sentence-level downstream tasks. It is tempting to apply these cross-lingual models to document representation learning. However, there are two challenges: (1) these models impose high costs on long document processing and thus many of them have strict length limit; (2) model fine-tuning requires extra data and computational resources, which is not practical in resource-limited settings. In this work, we address these challenges by proposing unsupervised Language-Agnostic Weighted Document Representations (LAWDR). We study the geometry of pre-trained sentence embeddings and leverage it to derive document representations without fine-tuning. Evaluated on cross-lingual document alignment, LAWDR demonstrates comparable performance to state-of-the-art models on benchmark datasets.