Are You Sure You're Positive? Consolidating Chain-of-Thought Agents with Uncertainty Quantification for Aspect-Category Sentiment Analysis
This addresses the challenge of expensive annotation and poor transferability in supervised methods for sentiment analysis, though it is incremental in leveraging existing LLM capabilities.
The paper tackles the problem of aspect-category sentiment analysis in label-scarce domains by proposing techniques that combine multiple chain-of-thought agents using large language models' token-level uncertainty scores, achieving results that demonstrate practical applicability with models like Llama and Qwen variants.
Aspect-category sentiment analysis provides granular insights by identifying specific themes within product reviews that are associated with particular opinions. Supervised learning approaches dominate the field. However, data is scarce and expensive to annotate for new domains. We argue that leveraging large language models in a zero-shot setting is beneficial where the time and resources required for dataset annotation are limited. Furthermore, annotation bias may lead to strong results using supervised methods but transfer poorly to new domains in contexts that lack annotations and demand reproducibility. In our work, we propose novel techniques that combine multiple chain-of-thought agents by leveraging large language models' token-level uncertainty scores. We experiment with the 3B and 70B+ parameter size variants of Llama and Qwen models, demonstrating how these approaches can fulfil practical needs and opening a discussion on how to gauge accuracy in label-scarce conditions.