Measuring Chain of Thought Faithfulness by Unlearning Reasoning Steps
This addresses the issue of verifying reasoning faithfulness in language models for researchers and practitioners, though it is incremental as it builds on existing CoT prompting work.
The authors tackled the problem of measuring whether language models' chain-of-thought reasoning reflects their true parametric beliefs, and introduced FUR, a framework that unlearns reasoning steps to assess faithfulness, showing it can precisely change model predictions in multi-hop question answering tasks.
When prompted to think step-by-step, language models (LMs) produce a chain of thought (CoT), a sequence of reasoning steps that the model supposedly used to produce its prediction. Despite much work on CoT prompting, it is unclear if reasoning verbalized in a CoT is faithful to the models' parametric beliefs. We introduce a framework for measuring parametric faithfulness of generated reasoning, and propose Faithfulness by Unlearning Reasoning steps (FUR), an instance of this framework. FUR erases information contained in reasoning steps from model parameters, and measures faithfulness as the resulting effect on the model's prediction. Our experiments with four LMs and five multi-hop multi-choice question answering (MCQA) datasets show that FUR is frequently able to precisely change the underlying models' prediction for a given instance by unlearning key steps, indicating when a CoT is parametrically faithful. Further analysis shows that CoTs generated by models post-unlearning support different answers, hinting at a deeper effect of unlearning.