Enhancing Robustness in Large Language Models: Prompting for Mitigating the Impact of Irrelevant Information
This addresses a robustness issue in LLMs for reasoning tasks, particularly in educational or real-world applications where irrelevant data is common, but it is incremental as it builds on existing prompting techniques.
The paper tackles the problem of large language models (LLMs) struggling with irrelevant information in reasoning tasks, showing that while LLMs can identify such information, they fail to mitigate its interference. The proposed ATF method significantly improves reasoning performance on the GSMIR dataset, a collection of primary school math problems with irrelevant information.
In recent years, Large language models (LLMs) have garnered significant attention due to their superior performance in complex reasoning tasks. However, recent studies may diminish their reasoning capabilities markedly when problem descriptions contain irrelevant information, even with the use of advanced prompting techniques. To further investigate this issue, a dataset of primary school mathematics problems containing irrelevant information, named GSMIR, was constructed. Testing prominent LLMs and prompting techniques on this dataset revealed that while LLMs can identify irrelevant information, they do not effectively mitigate the interference it causes once identified. A novel automatic construction method, ATF, which enhances the ability of LLMs to identify and self-mitigate the influence of irrelevant information, is proposed to address this shortcoming. This method operates in two steps: first, analysis of irrelevant information, followed by its filtering. The ATF method, as demonstrated by experimental results, significantly improves the reasoning performance of LLMs and prompting techniques, even in the presence of irrelevant information on the GSMIR dataset.