TR-BERT: Dynamic Token Reduction for Accelerating BERT Inference
This addresses the problem of slow inference for resource-limited applications in NLP, offering a method that is incremental but provides practical speed gains.
The paper tackles the computational expense of pre-trained language models in inference by proposing TR-BERT, a dynamic token reduction approach that accelerates BERT by 2-5 times on various NLP tasks while maintaining or improving performance, especially in long-text scenarios.
Existing pre-trained language models (PLMs) are often computationally expensive in inference, making them impractical in various resource-limited real-world applications. To address this issue, we propose a dynamic token reduction approach to accelerate PLMs' inference, named TR-BERT, which could flexibly adapt the layer number of each token in inference to avoid redundant calculation. Specially, TR-BERT formulates the token reduction process as a multi-step token selection problem and automatically learns the selection strategy via reinforcement learning. The experimental results on several downstream NLP tasks show that TR-BERT is able to speed up BERT by 2-5 times to satisfy various performance demands. Moreover, TR-BERT can also achieve better performance with less computation in a suite of long-text tasks since its token-level layer number adaption greatly accelerates the self-attention operation in PLMs. The source code and experiment details of this paper can be obtained from https://github.com/thunlp/TR-BERT.