The Sufficiency-Conciseness Trade-off in LLM Self-Explanation from an Information Bottleneck Perspective
This addresses the cost and efficiency problem for users of large language models by optimizing explanation generation, though it is incremental in applying information bottleneck principles to a known bottleneck.
The paper tackles the trade-off between explanation sufficiency and conciseness in LLM self-explanations, showing that more concise explanations often maintain accuracy while reducing length, with experiments on the ARC Challenge dataset in English and Persian.
Large Language Models increasingly rely on self-explanations, such as chain of thought reasoning, to improve performance on multi step question answering. While these explanations enhance accuracy, they are often verbose and costly to generate, raising the question of how much explanation is truly necessary. In this paper, we examine the trade-off between sufficiency, defined as the ability of an explanation to justify the correct answer, and conciseness, defined as the reduction in explanation length. Building on the information bottleneck principle, we conceptualize explanations as compressed representations that retain only the information essential for producing correct answers.To operationalize this view, we introduce an evaluation pipeline that constrains explanation length and assesses sufficiency using multiple language models on the ARC Challenge dataset. To broaden the scope, we conduct experiments in both English, using the original dataset, and Persian, as a resource-limited language through translation. Our experiments show that more concise explanations often remain sufficient, preserving accuracy while substantially reducing explanation length, whereas excessive compression leads to performance degradation.