TELeR: A General Taxonomy of LLM Prompts for Benchmarking Complex Tasks
This work addresses a methodological gap for researchers conducting benchmarking studies on complex tasks with LLMs, though it is incremental as it builds on existing prompt design practices.
The paper tackles the challenge of benchmarking LLMs on complex tasks by proposing TELeR, a general taxonomy for designing prompts with specific properties, enabling meaningful comparisons across studies and more accurate conclusions about LLM performance.
While LLMs have shown great success in understanding and generating text in traditional conversational settings, their potential for performing ill-defined complex tasks is largely under-studied. Indeed, we are yet to conduct comprehensive benchmarking studies with multiple LLMs that are exclusively focused on a complex task. However, conducting such benchmarking studies is challenging because of the large variations in LLMs' performance when different prompt types/styles are used and different degrees of detail are provided in the prompts. To address this issue, the paper proposes a general taxonomy that can be used to design prompts with specific properties in order to perform a wide range of complex tasks. This taxonomy will allow future benchmarking studies to report the specific categories of prompts used as part of the study, enabling meaningful comparisons across different studies. Also, by establishing a common standard through this taxonomy, researchers will be able to draw more accurate conclusions about LLMs' performance on a specific complex task.