Confidence Optimization for Probabilistic Encoding
This work addresses a specific bottleneck in probabilistic encoding for NLP classification, offering incremental improvements to enhance reliability and generalization.
The paper tackles the problem of Gaussian noise distorting distance measurements in probabilistic encoding for neural networks, proposing a confidence optimization method that improves performance and generalization on BERT and RoBERTa models in natural language classification tasks.
Probabilistic encoding introduces Gaussian noise into neural networks, enabling a smooth transition from deterministic to uncertain states and enhancing generalization ability. However, the randomness of Gaussian noise distorts point-based distance measurements in classification tasks. To mitigate this issue, we propose a confidence optimization probabilistic encoding (CPE) method that improves distance reliability and enhances representation learning. Specifically, we refine probabilistic encoding with two key strategies: First, we introduce a confidence-aware mechanism to adjust distance calculations, ensuring consistency and reliability in probabilistic encoding classification tasks. Second, we replace the conventional KL divergence-based variance regularization, which relies on unreliable prior assumptions, with a simpler L2 regularization term to directly constrain variance. The method we proposed is model-agnostic, and extensive experiments on natural language classification tasks demonstrate that our method significantly improves performance and generalization on both the BERT and the RoBERTa model.