To Predict or Not to Predict? Towards reliable uncertainty estimation in the presence of noise
This research provides actionable insights for developers of multilingual NLP systems to improve reliability in real-world, noisy environments, particularly for non-topical classification tasks.
This study investigates uncertainty estimation (UE) methods in multilingual text classification, specifically for complex-vs-simple sentence classification under noisy and non-topical conditions. While softmax-based methods decline in low-resource or domain-shift scenarios, Monte Carlo dropout approaches consistently perform well across languages, improving macro F1 from 0.81 to 0.85 when abstaining from 10% most uncertain instances in a non-topical classification task.
This study examines the role of uncertainty estimation (UE) methods in multilingual text classification under noisy and non-topical conditions. Using a complex-vs-simple sentence classification task across several languages, we evaluate a range of UE techniques against a range of metrics to assess their contribution to making more robust predictions. Results indicate that while methods relying on softmax outputs remain competitive in high-resource in-domain settings, their reliability declines in low-resource or domain-shift scenarios. In contrast, Monte Carlo dropout approaches demonstrate consistently strong performance across all languages, offering more robust calibration, stable decision thresholds, and greater discriminative power even under adverse conditions. We further demonstrate the positive impact of UE on non-topical classification: abstaining from predicting the 10\% most uncertain instances increases the macro F1 score from 0.81 to 0.85 in the Readme task. By integrating UE with trustworthiness metrics, this study provides actionable insights for developing more reliable NLP systems in real-world multilingual environments. See https://github.com/Nouran-Khallaf/To-Predict-or-Not-to-Predict