Learn to Accumulate Evidence from All Training Samples: Theory and Practice
This work addresses a fundamental limitation in evidential deep learning, improving uncertainty quantification for researchers and practitioners, though it is incremental as it builds on existing evidential models.
The paper tackled the problem of evidential deep learning models having inferior predictive performance due to zero evidence regions created by activation functions, and proposed a novel regularizer that effectively alleviates this limitation, as confirmed by extensive experiments on challenging datasets.
Evidential deep learning, built upon belief theory and subjective logic, offers a principled and computationally efficient way to turn a deterministic neural network uncertainty-aware. The resultant evidential models can quantify fine-grained uncertainty using the learned evidence. To ensure theoretically sound evidential models, the evidence needs to be non-negative, which requires special activation functions for model training and inference. This constraint often leads to inferior predictive performance compared to standard softmax models, making it challenging to extend them to many large-scale datasets. To unveil the real cause of this undesired behavior, we theoretically investigate evidential models and identify a fundamental limitation that explains the inferior performance: existing evidential activation functions create zero evidence regions, which prevent the model to learn from training samples falling into such regions. A deeper analysis of evidential activation functions based on our theoretical underpinning inspires the design of a novel regularizer that effectively alleviates this fundamental limitation. Extensive experiments over many challenging real-world datasets and settings confirm our theoretical findings and demonstrate the effectiveness of our proposed approach.