Mothman at SemEval-2024 Task 9: An Iterative System for Chain-of-Thought Prompt Optimization
This addresses a specific challenge in NLP for researchers and practitioners by improving model performance on creative, logic-based tasks, though it is incremental as it builds on existing prompt engineering methods.
The paper tackled the problem of poor performance of large language models on lateral thinking tasks like BrainTeaser by proposing an iterative chain-of-thought prompt optimization system, resulting in significant performance improvements as demonstrated on the SemEval-2024 shared task.
Extensive research exists on the performance of large language models on logic-based tasks, whereas relatively little has been done on their ability to generate creative solutions on lateral thinking tasks. The BrainTeaser shared task tests lateral thinking and uses adversarial datasets to prevent memorization, resulting in poor performance for out-of-the-box models. We propose a system for iterative, chain-of-thought prompt engineering which optimizes prompts using human evaluation. Using this shared task, we demonstrate our system's ability to significantly improve model performance by optimizing prompts and evaluate the input dataset.