Adapter Pruning using Tropical Characterization
This work addresses the need for efficient adapter pruning in NLP, offering a method that improves parameter relevance identification, though it is incremental as it builds on existing adapter techniques.
The paper tackles the problem of determining the optimal number of parameters for adapters in transfer learning by proposing an adapter pruning method based on tropical geometry, which identifies more relevant parameters to prune compared to a magnitude-based baseline, with a combined approach performing best across five NLP datasets.
Adapters are widely popular parameter-efficient transfer learning approaches in natural language processing that insert trainable modules in between layers of a pre-trained language model. Apart from several heuristics, however, there has been a lack of studies analyzing the optimal number of adapter parameters needed for downstream applications. In this paper, we propose an adapter pruning approach by studying the tropical characteristics of trainable modules. We cast it as an optimization problem that aims to prune parameters from the adapter layers without changing the orientation of underlying tropical hypersurfaces. Our experiments on five NLP datasets show that tropical geometry tends to identify more relevant parameters to prune when compared with the magnitude-based baseline, while a combined approach works best across the tasks.