Multilingual Extraction and Recognition of Implicit Discourse Relations in Speech and Text
This work addresses the problem of understanding implicit discourse relations in speech and text for multilingual applications, representing an incremental advance by combining existing methods for multimodal and cross-lingual learning.
The paper tackled the challenge of classifying implicit discourse relations across languages and modalities by constructing a multilingual and multimodal dataset for English, French, and Spanish, and found that integrating textual and acoustic information improved performance, with cross-lingual transfer benefiting low-resource languages.
Implicit discourse relation classification is a challenging task, as it requires inferring meaning from context. While contextual cues can be distributed across modalities and vary across languages, they are not always captured by text alone. To address this, we introduce an automatic method for distantly related and unrelated language pairs to construct a multilingual and multimodal dataset for implicit discourse relations in English, French, and Spanish. For classification, we propose a multimodal approach that integrates textual and acoustic information through Qwen2-Audio, allowing joint modeling of text and audio for implicit discourse relation classification across languages. We find that while text-based models outperform audio-based models, integrating both modalities can enhance performance, and cross-lingual transfer can provide substantial improvements for low-resource languages.