Classifying Scientific Publications with BERT -- Is Self-Attention a Feature Selection Method?
This work addresses the interpretability and optimization of transformer models for text classification, though it is incremental in analyzing existing mechanisms.
The study investigated whether BERT's self-attention mechanism acts as a feature selection method for classifying scientific articles by discipline, finding that it focuses on domain-relevant words but is outperformed by conventional feature selection methods for learning classifiers from scratch.
We investigate the self-attention mechanism of BERT in a fine-tuning scenario for the classification of scientific articles over a taxonomy of research disciplines. We observe how self-attention focuses on words that are highly related to the domain of the article. Particularly, a small subset of vocabulary words tends to receive most of the attention. We compare and evaluate the subset of the most attended words with feature selection methods normally used for text classification in order to characterize self-attention as a possible feature selection approach. Using ConceptNet as ground truth, we also find that attended words are more related to the research fields of the articles. However, conventional feature selection methods are still a better option to learn classifiers from scratch. This result suggests that, while self-attention identifies domain-relevant terms, the discriminatory information in BERT is encoded in the contextualized outputs and the classification layer. It also raises the question whether injecting feature selection methods in the self-attention mechanism could further optimize single sequence classification using transformers.