Fast Multi-Resolution Transformer Fine-tuning for Extreme Multi-label Text Classification
This work addresses a computational bottleneck for researchers and practitioners in XMC applications like recommendation systems and document tagging, offering a significant speed improvement with better accuracy, though it is incremental as it builds on existing transformer-based methods.
The paper tackles the lengthy computational time of fine-tuning transformer models for extreme multi-label text classification (XMC) by proposing XR-Transformer, a recursive approach that accelerates training while improving performance, achieving a 20x speedup and increasing Precision@1 from 51% to 54% on the Amazon-3M dataset.
Extreme multi-label text classification (XMC) seeks to find relevant labels from an extreme large label collection for a given text input. Many real-world applications can be formulated as XMC problems, such as recommendation systems, document tagging and semantic search. Recently, transformer based XMC methods, such as X-Transformer and LightXML, have shown significant improvement over other XMC methods. Despite leveraging pre-trained transformer models for text representation, the fine-tuning procedure of transformer models on large label space still has lengthy computational time even with powerful GPUs. In this paper, we propose a novel recursive approach, XR-Transformer to accelerate the procedure through recursively fine-tuning transformer models on a series of multi-resolution objectives related to the original XMC objective function. Empirical results show that XR-Transformer takes significantly less training time compared to other transformer-based XMC models while yielding better state-of-the-art results. In particular, on the public Amazon-3M dataset with 3 million labels, XR-Transformer is not only 20x faster than X-Transformer but also improves the Precision@1 from 51% to 54%.