How Out-of-Distribution Detection Learning Theory Enhances Transformer: Learnability and Reliability
It addresses the reliability issue for transformers in real-world applications where data distributions may shift, though it appears incremental by building on existing OOD detection concepts.
This paper tackles the problem of transformers struggling to generalize to out-of-distribution (OOD) data by introducing a PAC theory for OOD detection, which establishes conditions for learnability and leads to a novel algorithm that achieves state-of-the-art performance across various data formats.
Transformers excel in natural language processing and computer vision tasks. However, they still face challenges in generalizing to Out-of-Distribution (OOD) datasets, i.e. data whose distribution differs from that seen during training. OOD detection aims to distinguish outliers while preserving in-distribution (ID) data performance. This paper introduces the OOD detection Probably Approximately Correct (PAC) Theory for transformers, which establishes the conditions for data distribution and model configurations for the OOD detection learnability of transformers. It shows that outliers can be accurately represented and distinguished with sufficient data under conditions. The theoretical implications highlight the trade-off between theoretical principles and practical training paradigms. By examining this trade-off, we naturally derived the rationale for leveraging auxiliary outliers to enhance OOD detection. Our theory suggests that by penalizing the misclassification of outliers within the loss function and strategically generating soft synthetic outliers, one can robustly bolster the reliability of transformer networks. This approach yields a novel algorithm that ensures learnability and refines the decision boundaries between inliers and outliers. In practice, the algorithm consistently achieves state-of-the-art (SOTA) performance across various data formats.