Is Stance Detection Topic-Independent and Cross-topic Generalizable? -- A Reproduction Study
This work highlights limitations in cross-topic generalization for stance detection, suggesting future research needs to address topic-specific vocabulary and context, but it is incremental as it builds on prior reproduction studies.
The study reproduced a state-of-the-art cross-topic stance detection model and analyzed its performance across unseen topics, finding that topic, class, and their interaction affect model performance, with specific topics like abortion and cloning showing variations.
Cross-topic stance detection is the task to automatically detect stances (pro, against, or neutral) on unseen topics. We successfully reproduce state-of-the-art cross-topic stance detection work (Reimers et. al., 2019), and systematically analyze its reproducibility. Our attention then turns to the cross-topic aspect of this work, and the specificity of topics in terms of vocabulary and socio-cultural context. We ask: To what extent is stance detection topic-independent and generalizable across topics? We compare the model's performance on various unseen topics, and find topic (e.g. abortion, cloning), class (e.g. pro, con), and their interaction affecting the model's performance. We conclude that investigating performance on different topics, and addressing topic-specific vocabulary and context, is a future avenue for cross-topic stance detection.