Multilingual and Continuous Backchannel Prediction: A Cross-lingual Study
This work addresses the need for more natural and culturally-aware spoken dialogue systems by providing a unified model and empirical evidence on cross-linguistic backchannel timing differences, though it is incremental as it builds on existing multilingual and Transformer-based approaches.
The researchers tackled the problem of predicting backchannels in multilingual spoken dialogue by developing a Transformer-based model for Japanese, English, and Chinese, which matched or surpassed monolingual baselines across all three languages. They found that the model learns both universal and language-specific timing patterns, with distinct cue usage across languages, such as Japanese relying more on short-term linguistic information and Chinese benefiting from longer contexts.
We present a multilingual, continuous backchannel prediction model for Japanese, English, and Chinese, and use it to investigate cross-linguistic timing behavior. The model is Transformer-based and operates at the frame level, jointly trained with auxiliary tasks on approximately 300 hours of dyadic conversations. Across all three languages, the multilingual model matches or surpasses monolingual baselines, indicating that it learns both language-universal cues and language-specific timing patterns. Zero-shot transfer with two-language training remains limited, underscoring substantive cross-lingual differences. Perturbation analyses reveal distinct cue usage: Japanese relies more on short-term linguistic information, whereas English and Chinese are more sensitive to silence duration and prosodic variation; multilingual training encourages shared yet adaptable representations and reduces overreliance on pitch in Chinese. A context-length study further shows that Japanese is relatively robust to shorter contexts, while Chinese benefits markedly from longer contexts. Finally, we integrate the trained model into a real-time processing software, demonstrating CPU-only inference. Together, these findings provide a unified model and empirical evidence for how backchannel timing differs across languages, informing the design of more natural, culturally-aware spoken dialogue systems.