Reassessing the Role of Chain-of-Thought in Sentiment Analysis: Insights and Limitations
This work addresses the philosophical debate on language and thought in AI, but it is incremental as it applies existing methods to a specific domain with limited practical gains.
The paper investigated whether reasoning techniques like chain-of-thought prompting improve semantic understanding in sentiment analysis, finding minimal impact as models focus on aspect terms rather than sentiment and rely on demonstration information.
The relationship between language and thought remains an unresolved philosophical issue. Existing viewpoints can be broadly categorized into two schools: one asserting their independence, and another arguing that language constrains thought. In the context of large language models, this debate raises a crucial question: Does a language model's grasp of semantic meaning depend on thought processes? To explore this issue, we investigate whether reasoning techniques can facilitate semantic understanding. Specifically, we conceptualize thought as reasoning, employ chain-of-thought prompting as a reasoning technique, and examine its impact on sentiment analysis tasks. The experiments show that chain-of-thought has a minimal impact on sentiment analysis tasks. Both the standard and chain-of-thought prompts focus on aspect terms rather than sentiment in the generated content. Furthermore, counterfactual experiments reveal that the model's handling of sentiment tasks primarily depends on information from demonstrations. The experimental results support the first viewpoint.