Contrastive Out-of-Distribution Detection for Pretrained Transformers
This addresses the issue of semantic shifts in real-world scenarios for users of pretrained Transformers, though it is incremental as it builds on existing contrastive learning and OOD detection methods.
The paper tackles the problem of out-of-distribution (OOD) detection for pretrained Transformers by fine-tuning them with a contrastive loss to improve representation compactness, achieving near-perfect OOD detection performance and outperforming baselines drastically.
Pretrained Transformers achieve remarkable performance when training and test data are from the same distribution. However, in real-world scenarios, the model often faces out-of-distribution (OOD) instances that can cause severe semantic shift problems at inference time. Therefore, in practice, a reliable model should identify such instances, and then either reject them during inference or pass them over to models that handle another distribution. In this paper, we develop an unsupervised OOD detection method, in which only the in-distribution (ID) data are used in training. We propose to fine-tune the Transformers with a contrastive loss, which improves the compactness of representations, such that OOD instances can be better differentiated from ID ones. These OOD instances can then be accurately detected using the Mahalanobis distance in the model's penultimate layer. We experiment with comprehensive settings and achieve near-perfect OOD detection performance, outperforming baselines drastically. We further investigate the rationales behind the improvement, finding that more compact representations through margin-based contrastive learning bring the improvement. We release our code to the community for future research.