VECHR: A Dataset for Explainable and Robust Classification of Vulnerability Type in the European Court of Human Rights
This addresses the need for explainable and robust NLP tools in legal contexts, specifically for human rights vulnerability assessment, but is incremental as it focuses on dataset creation and benchmarking without major methodological breakthroughs.
The paper tackles the problem of classifying vulnerability types in European Court of Human Rights cases by introducing VECHR, a novel expert-annotated dataset, and benchmarks state-of-the-art models, showing lower prediction performance and limited robustness with out-of-domain data.
Recognizing vulnerability is crucial for understanding and implementing targeted support to empower individuals in need. This is especially important at the European Court of Human Rights (ECtHR), where the court adapts Convention standards to meet actual individual needs and thus ensures effective human rights protection. However, the concept of vulnerability remains elusive at the ECtHR and no prior NLP research has dealt with it. To enable future research in this area, we present VECHR, a novel expert-annotated multi-label dataset comprising of vulnerability type classification and explanation rationale. We benchmark the performance of state-of-the-art models on VECHR from both prediction and explainability perspectives. Our results demonstrate the challenging nature of the task with lower prediction performance and limited agreement between models and experts. Further, we analyze the robustness of these models in dealing with out-of-domain (OOD) data and observe overall limited performance. Our dataset poses unique challenges offering significant room for improvement regarding performance, explainability, and robustness.