Granting GPT-4 License and Opportunity: Enhancing Accuracy and Confidence Estimation for Few-Shot Event Detection
This addresses the issue of confidence estimation for users of GPT-4 in few-shot learning tasks like event detection, though it is incremental as it builds on existing prompt engineering without new computational machinery.
The paper tackled the problem of unreliable confidence estimation in GPT-4 for few-shot event detection by introducing a prompt-based method called License and Opportunity (L&O), which improved accuracy and achieved a usable confidence measure with an AUC of 0.759.
Large Language Models (LLMs) such as GPT-4 have shown enough promise in the few-shot learning context to suggest use in the generation of "silver" data and refinement of new ontologies through iterative application and review. Such workflows become more effective with reliable confidence estimation. Unfortunately, confidence estimation is a documented weakness of models such as GPT-4, and established methods to compensate require significant additional complexity and computation. The present effort explores methods for effective confidence estimation with GPT-4 with few-shot learning for event detection in the BETTER ontology as a vehicle. The key innovation is expanding the prompt and task presented to GPT-4 to provide License to speculate when unsure and Opportunity to quantify and explain its uncertainty (L&O). This approach improves accuracy and provides usable confidence measures (0.759 AUC) with no additional machinery.