Zero-Shot Belief: A Hard Problem for LLMs
This work addresses a hard problem for large language models, particularly in the context of natural language processing and belief prediction tasks.
The authors tackled the problem of zero-shot source-and-target belief prediction, achieving new state-of-the-art results on FactBank with a hybrid approach. The results show that multiple LLMs struggle with the task, but the hybrid approach achieves SOTA performance.
We present two LLM-based approaches to zero-shot source-and-target belief prediction on FactBank: a unified system that identifies events, sources, and belief labels in a single pass, and a hybrid approach that uses a fine-tuned DeBERTa tagger for event detection. We show that multiple open-sourced, closed-source, and reasoning-based LLMs struggle with the task. Using the hybrid approach, we achieve new state-of-the-art results on FactBank and offer a detailed error analysis. Our approach is then tested on the Italian belief corpus ModaFact.