Discovering the Rationale of Decisions: Experiments on Aligning Learning and Reasoning
This work addresses the need for explainable and fair decision support systems in legal settings, though it appears incremental as it builds on existing concepts like unit-testing.
The paper tackles the problem of evaluating and improving the rationale behind black-box machine learning systems in AI and law, showing that a knowledge-driven method can analyze which rationale elements are learned and adjust them using tailor-made training data.
In AI and law, systems that are designed for decision support should be explainable when pursuing justice. In order for these systems to be fair and responsible, they should make correct decisions and make them using a sound and transparent rationale. In this paper, we introduce a knowledge-driven method for model-agnostic rationale evaluation using dedicated test cases, similar to unit-testing in professional software development. We apply this new method in a set of machine learning experiments aimed at extracting known knowledge structures from artificial datasets from fictional and non-fictional legal settings. We show that our method allows us to analyze the rationale of black-box machine learning systems by assessing which rationale elements are learned or not. Furthermore, we show that the rationale can be adjusted using tailor-made training data based on the results of the rationale evaluation.