Rationales Are Not Silver Bullets: Measuring the Impact of Rationales on Model Performance and Reliability
This work challenges the prevailing view that rationales always benefit models, providing critical insights for researchers and practitioners in AI and NLP on their use and alignment with human reasoning.
The paper investigates the impact of rationales on language model performance and reliability, finding that rationales can sometimes degrade performance but improve reliability, with a linear relationship between these effects driven by task difficulty.
Training language models with rationales augmentation has been shown to be beneficial in many existing works. In this paper, we identify that such a prevailing view does not hold consistently. We conduct comprehensive investigations to thoroughly inspect the impact of rationales on model performance as well as a novel perspective of model reliability. The results lead to several key findings that add new insights upon existing understandings: 1) Rationales can, at times, deteriorate model performance; 2) Rationales can, at times, improve model reliability, even outperforming their untrained counterparts; 3) A linear correspondence exists in between the performance and reliability improvements, while both are driven by the intrinsic difficulty of the task. These findings provide informative regulations on the broad utilization of rationales and raise critical implications on the procedure of explicitly aligning language models with implicit human thoughts. Codes can be found at https://github.com/Ignoramus0817/rationales.