Causal Distillation: Transferring Structured Explanations from Large to Compact Language Models
This addresses the challenge of enabling compact models to perform robust causal reasoning, which is incremental as it builds on existing distillation techniques but focuses specifically on structured explanations.
The paper tackles the problem of transferring causal reasoning abilities from large proprietary language models to smaller open-source models by introducing a framework for distilling structured cause-and-effect explanations, resulting in a new metric called Causal Explanation Coherence (CEC) to evaluate explanation quality.
Large proprietary language models exhibit strong causal reasoning abilities that smaller open-source models struggle to replicate. We introduce a novel framework for distilling causal explanations that transfers causal reasoning skills from a powerful teacher model to a compact open-source model. The key idea is to train the smaller model to develop causal reasoning abilities by generating structured cause-and-effect explanations consistent with those of the teacher model. To evaluate the quality of the student-generated explanations, we introduce a new metric called Causal Explanation Coherence (CEC) to assess the structural and logical consistency of causal reasoning. This metric uses sentence-level semantic alignment to measure how well each part of the generated explanation corresponds to the teacher's reference, capturing both faithfulness and coverage of the underlying causal chain. Our framework and the CEC metric provide a principled foundation for training smaller models to perform robust causal reasoning and for systematically assessing the coherence of explanations in language model outputs.