Misaligning Reasoning with Answers -- A Framework for Assessing LLM CoT Robustness
This addresses the need for trustworthy LLM reasoning in domains like education and healthcare, though it is incremental as it builds on existing explanation techniques.
The paper tackles the problem of opaque decision-making in LLMs by introducing MATCHA, a framework to assess the robustness of Chain-of-Thought reasoning under input perturbations, revealing that LLMs are more vulnerable in multi-step and commonsense tasks than logical ones, with non-trivial transfer rates to black-box models.
LLMs' decision-making process is opaque, prompting the need for explanation techniques like Chain-of-Thought. To investigate the relationship between answer and reasoning, we design a novel evaluation framework, MATCHA. In domains like education and healthcare, reasoning is key for model trustworthiness. MATCHA reveals that LLMs under input perturbations can give inconsistent or nonsensical reasoning. Additionally, we use LLM judges to assess reasoning robustness across models. Our results show that LLMs exhibit greater vulnerability to input perturbations for multi-step and commonsense tasks than compared to logical tasks. Also, we show non-trivial transfer rates of our successful examples to black-box models. Our evaluation framework helps to better understand LLM reasoning mechanisms and guides future models toward more robust and reasoning-driven architectures, enforcing answer-reasoning consistency.