Reliable Classification with Conformal Learning and Interval-Type 2 Fuzzy Sets
This work addresses reliability assessment for deployed machine learning systems, offering an incremental improvement over existing conformal and fuzzy methods.
The paper tackles the problem of unreliable classifier confidence by combining conformal learning with fuzzy rule-based systems, showing that type 2 fuzzy sets improve output quality compared to fuzzy and crisp rules, with performance metrics provided.
Classical machine learning classifiers tend to be overconfident can be unreliable outside of the laboratory benchmarks. Properly assessing the reliability of the output of the model per sample is instrumental for real-life scenarios where these systems are deployed. Because of this, different techniques have been employed to properly quantify the quality of prediction for a given model. These are most commonly Bayesian statistics and, more recently, conformal learning. Given a calibration set, conformal learning can produce outputs that are guaranteed to cover the target class with a desired significance level, and are more reliable than the standard confidence intervals used by Bayesian methods. In this work, we propose to use conformal learning with fuzzy rule-based systems in classification and show some metrics of their performance. Then, we discuss how the use of type 2 fuzzy sets can improve the quality of the output of the system compared to both fuzzy and crisp rules. Finally, we also discuss how the fine-tuning of the system can be adapted to improve the quality of the conformal prediction.