Picturized and Recited with Dialects: A Multimodal Chinese Representation Framework for Sentiment Analysis of Classical Chinese Poetry
This work addresses the problem of analyzing sentiment in classical Chinese poetry for researchers and cultural applications by integrating multimodal features, though it is incremental as it builds on existing multimodal methods.
The authors tackled sentiment analysis of classical Chinese poetry by proposing a dialect-enhanced multimodal framework that incorporates audio from multiple dialects and visual features, achieving at least 2.51% improvement in accuracy and 1.63% in macro F1 over state-of-the-art methods on two public datasets.
Classical Chinese poetry is a vital and enduring part of Chinese literature, conveying profound emotional resonance. Existing studies analyze sentiment based on textual meanings, overlooking the unique rhythmic and visual features inherent in poetry,especially since it is often recited and accompanied by Chinese paintings. In this work, we propose a dialect-enhanced multimodal framework for classical Chinese poetry sentiment analysis. We extract sentence-level audio features from the poetry and incorporate audio from multiple dialects,which may retain regional ancient Chinese phonetic features, enriching the phonetic representation. Additionally, we generate sentence-level visual features, and the multimodal features are fused with textual features enhanced by LLM translation through multimodal contrastive representation learning. Our framework outperforms state-of-the-art methods on two public datasets, achieving at least 2.51% improvement in accuracy and 1.63% in macro F1. We open-source the code to facilitate research in this area and provide insights for general multimodal Chinese representation.