LS-Tree: Model Interpretation When the Data Are Linguistic
This work addresses interpretability for users of language models, though it appears incremental as it builds on existing parse tree and game theory concepts.
The authors tackled the problem of interpreting trained classification models for linguistic data by proposing LS-Tree, a method that assigns importance scores to words using parse trees and syntactic structure, and demonstrated its utility in aiding interpretability and diagnostics for several language models.
We study the problem of interpreting trained classification models in the setting of linguistic data sets. Leveraging a parse tree, we propose to assign least-squares based importance scores to each word of an instance by exploiting syntactic constituency structure. We establish an axiomatic characterization of these importance scores by relating them to the Banzhaf value in coalitional game theory. Based on these importance scores, we develop a principled method for detecting and quantifying interactions between words in a sentence. We demonstrate that the proposed method can aid in interpretability and diagnostics for several widely-used language models.