Uncertainty Toolbox: an Open-Source Library for Assessing, Visualizing, and Improving Uncertainty Quantification
This work addresses the problem of disjoint and incomparable UQ metrics for researchers and practitioners in machine learning, though it is incremental as it builds on existing UQ methods without introducing new ones.
The authors tackled the lack of standardized evaluation and implementation for uncertainty quantification (UQ) in machine learning by introducing Uncertainty Toolbox, an open-source Python library that provides tools for assessing, visualizing, and improving UQ, along with educational resources.
With increasing deployment of machine learning systems in various real-world tasks, there is a greater need for accurate quantification of predictive uncertainty. While the common goal in uncertainty quantification (UQ) in machine learning is to approximate the true distribution of the target data, many works in UQ tend to be disjoint in the evaluation metrics utilized, and disparate implementations for each metric lead to numerical results that are not directly comparable across different works. To address this, we introduce Uncertainty Toolbox, an open-source python library that helps to assess, visualize, and improve UQ. Uncertainty Toolbox additionally provides pedagogical resources, such as a glossary of key terms and an organized collection of key paper references. We hope that this toolbox is useful for accelerating and uniting research efforts in uncertainty in machine learning.