Are Structural Concepts Universal in Transformer Language Models? Towards Interpretable Cross-Lingual Generalization
This addresses the challenge of improving cross-lingual generalization for low-resource languages in NLP, representing an incremental advance with a novel method for a known bottleneck.
The paper tackles the problem of unequal cross-lingual generalization in large language models, especially for low-resource languages, by investigating explicit alignment of structural concepts across languages; it finds high alignability in 43 languages and proposes a meta-learning method that achieves competitive results on syntactic analysis tasks, narrowing performance gaps.
Large language models (LLMs) have exhibited considerable cross-lingual generalization abilities, whereby they implicitly transfer knowledge across languages. However, the transfer is not equally successful for all languages, especially for low-resource ones, which poses an ongoing challenge. It is unclear whether we have reached the limits of implicit cross-lingual generalization and if explicit knowledge transfer is viable. In this paper, we investigate the potential for explicitly aligning conceptual correspondence between languages to enhance cross-lingual generalization. Using the syntactic aspect of language as a testbed, our analyses of 43 languages reveal a high degree of alignability among the spaces of structural concepts within each language for both encoder-only and decoder-only LLMs. We then propose a meta-learning-based method to learn to align conceptual spaces of different languages, which facilitates zero-shot and few-shot generalization in concept classification and also offers insights into the cross-lingual in-context learning phenomenon. Experiments on syntactic analysis tasks show that our approach achieves competitive results with state-of-the-art methods and narrows the performance gap between languages, particularly benefiting those with limited resources.