PPTC-R benchmark: Towards Evaluating the Robustness of Large Language Models for PowerPoint Task Completion
This work addresses the need for robust LLMs in real-world task completion for users relying on AI assistants, though it is incremental as it introduces a new benchmark rather than a novel method.
The authors tackled the problem of evaluating the robustness of Large Language Models (LLMs) in completing PowerPoint tasks by proposing the PPTC-R benchmark, which tests LLMs against adversarial instructions and software version variations, finding that GPT-4 performed best but all models struggled with multiple simultaneous challenges.
The growing dependence on Large Language Models (LLMs) for finishing user instructions necessitates a comprehensive understanding of their robustness to complex task completion in real-world situations. To address this critical need, we propose the PowerPoint Task Completion Robustness benchmark (PPTC-R) to measure LLMs' robustness to the user PPT task instruction and software version. Specifically, we construct adversarial user instructions by attacking user instructions at sentence, semantic, and multi-language levels. To assess the robustness of Language Models to software versions, we vary the number of provided APIs to simulate both the newest version and earlier version settings. Subsequently, we test 3 closed-source and 4 open-source LLMs using a benchmark that incorporates these robustness settings, aiming to evaluate how deviations impact LLMs' API calls for task completion. We find that GPT-4 exhibits the highest performance and strong robustness in our benchmark, particularly in the version update and the multilingual settings. However, we find that all LLMs lose their robustness when confronted with multiple challenges (e.g., multi-turn) simultaneously, leading to significant performance drops. We further analyze the robustness behavior and error reasons of LLMs in our benchmark, which provide valuable insights for researchers to understand the LLM's robustness in task completion and develop more robust LLMs and agents. We release the code and data at \url{https://github.com/ZekaiGalaxy/PPTCR}.