Betti numbers of attention graphs is all you really need
This provides a novel topological approach for analyzing attention mechanisms in NLP, though it appears incremental as it applies existing topological methods to a new context.
The researchers tackled the problem of analyzing attention-based neural networks by applying topological methods to attention graphs from BERT, showing that a classifier using Betti numbers achieves classification results comparable to conventional methods on three text classification benchmarks.
We apply methods of topological analysis to the attention graphs, calculated on the attention heads of the BERT model ( arXiv:1810.04805v2 ). Our research shows that the classifier built upon basic persistent topological features (namely, Betti numbers) of the trained neural network can achieve classification results on par with the conventional classification method. We show the relevance of such topological text representation on three text classification benchmarks. For the best of our knowledge, it is the first attempt to analyze the topology of an attention-based neural network, widely used for Natural Language Processing.