Can persistent homology whiten Transformer-based black-box models? A case study on BERT compression
This addresses the need for more efficient and interpretable large language models, particularly for deployment on resource-constrained devices, though it is incremental as it builds on existing compression and explainability techniques.
The paper tackles the problem of high computational costs and lack of interpretability in BERT models by proposing a method using persistent homology to explain and compress them, achieving parameter reductions of 58.47% for BERT Base and 52.3% for BERT Large while maintaining performance on the GLUE benchmark.
Large Language Models (LLMs) like BERT have gained significant prominence due to their remarkable performance in various natural language processing tasks. However, they come with substantial computational and memory costs. Additionally, they are essentially black-box models, challenging to explain and interpret. In this article, we propose Optimus BERT Compression and Explainability (OBCE), a methodology to bring explainability to BERT models using persistent homology, aiming to measure the importance of each neuron by studying the topological characteristics of their outputs. As a result, we can compress BERT significantly by reducing the number of parameters (58.47% of the original parameters for BERT Base, 52.3% for BERT Large). We evaluated our methodology on the standard GLUE Benchmark, comparing the results with state-of-the-art techniques and achieving outstanding results. Consequently, our methodology can "whiten" BERT models by providing explainability to its neurons and reducing the model's size, making it more suitable for deployment on resource-constrained devices.