Tricks and Plug-ins for Gradient Boosting with Transformers
This work addresses efficiency and performance issues for NLP practitioners using transformers, though it is incremental as it builds on existing boosting and transformer methods.
The paper tackles the computational and tuning challenges of transformers by proposing BoostTransformer, which integrates boosting principles with subgrid token selection and importance-weighted sampling, resulting in faster convergence and higher accuracy on fine-grained text classification benchmarks.
Transformer architectures dominate modern NLP but often demand heavy computational resources and intricate hyperparameter tuning. To mitigate these challenges, we propose a novel framework, BoostTransformer, that augments transformers with boosting principles through subgrid token selection and importance-weighted sampling. Our method incorporates a least square boosting objective directly into the transformer pipeline, enabling more efficient training and improved performance. Across multiple fine-grained text classification benchmarks, BoostTransformer demonstrates both faster convergence and higher accuracy, surpassing standard transformers while minimizing architectural search overhead.