Label Indeterminacy in AI & Law
This addresses a critical issue for AI applications in law by highlighting how unreliable labels can impact model performance, though it is incremental as it builds on existing concerns without introducing a new solution.
The paper tackles the problem of label indeterminacy in legal machine learning, where past case outcomes used as ground truth may be unreliable due to human interventions, and shows that different label constructions during training significantly affect model behavior in classifying cases from the European Court of Human Rights.
Machine learning is increasingly used in the legal domain, where it typically operates retrospectively by treating past case outcomes as ground truth. However, legal outcomes are often shaped by human interventions that are not captured in most machine learning approaches. A final decision may result from a settlement, an appeal, or other procedural actions. This creates label indeterminacy: the outcome could have been different if the intervention had or had not taken place. We argue that legal machine learning applications need to account for label indeterminacy. Methods exist that can impute these indeterminate labels, but they are all grounded in unverifiable assumptions. In the context of classifying cases from the European Court of Human Rights, we show that the way that labels are constructed during training can significantly affect model behaviour. We therefore position label indeterminacy as a relevant concern in AI & Law and demonstrate how it can shape model behaviour.