xInv: Explainable Optimization of Inverse Problems
This addresses the need for interpretability in inverse problems for fields like healthcare and climate science, though it appears incremental as it builds on existing explainability techniques.
The paper tackles the problem of making iterative optimization in inverse problems explainable to domain experts by proposing a method that instruments a differentiable simulator to emit natural language events and uses a Language Model to generate explanations from these traces, demonstrating effectiveness with an illustrative optimization problem and a neural network training example.
Inverse problems are central to a wide range of fields, including healthcare, climate science, and agriculture. They involve the estimation of inputs, typically via iterative optimization, to some known forward model so that it produces a desired outcome. Despite considerable development in the explainability and interpretability of forward models, the iterative optimization of inverse problems remains largely cryptic to domain experts. We propose a methodology to produce explanations, from traces produced by an optimizer, that are interpretable by humans at the abstraction of the domain. The central idea in our approach is to instrument a differentiable simulator so that it emits natural language events during its forward and backward passes. In a post-process, we use a Language Model to create an explanation from the list of events. We demonstrate the effectiveness of our approach with an illustrative optimization problem and an example involving the training of a neural network.