Protoformer: Embedding Prototypes for Transformers
This addresses robustness issues in text classification for real-world applications, but appears incremental as it builds on existing Transformer methods.
The paper tackled the problem of anomalies and noisy labels in text classification with Transformers by proposing Protoformer, a self-learning framework that leverages problematic samples, and demonstrated improved performance across diverse datasets and models.
Transformers have been widely applied in text classification. Unfortunately, real-world data contain anomalies and noisy labels that cause challenges for state-of-art Transformers. This paper proposes Protoformer, a novel self-learning framework for Transformers that can leverage problematic samples for text classification. Protoformer features a selection mechanism for embedding samples that allows us to efficiently extract and utilize anomalies prototypes and difficult class prototypes. We demonstrated such capabilities on datasets with diverse textual structures (e.g., Twitter, IMDB, ArXiv). We also applied the framework to several models. The results indicate that Protoformer can improve current Transformers in various empirical settings.