Large Language Models for Imbalanced Classification: Diversity makes the difference
This addresses the issue of reduced robustness in classification for practitioners dealing with imbalanced data, though it is incremental as it builds on existing LLM-based methods.
The paper tackles the problem of limited diversity in LLM-based oversampling for imbalanced classification by proposing a method that conditions generation on labels and features, uses permutation fine-tuning, and includes interpolated samples, resulting in significant outperformance over eight SOTA baselines across 10 tabular datasets.
Oversampling is one of the most widely used approaches for addressing imbalanced classification. The core idea is to generate additional minority samples to rebalance the dataset. Most existing methods, such as SMOTE, require converting categorical variables into numerical vectors, which often leads to information loss. Recently, large language model (LLM)-based methods have been introduced to overcome this limitation. However, current LLM-based approaches typically generate minority samples with limited diversity, reducing robustness and generalizability in downstream classification tasks. To address this gap, we propose a novel LLM-based oversampling method designed to enhance diversity. First, we introduce a sampling strategy that conditions synthetic sample generation on both minority labels and features. Second, we develop a new permutation strategy for fine-tuning pre-trained LLMs. Third, we fine-tune the LLM not only on minority samples but also on interpolated samples to further enrich variability. Extensive experiments on 10 tabular datasets demonstrate that our method significantly outperforms eight SOTA baselines. The generated synthetic samples are both realistic and diverse. Moreover, we provide theoretical analysis through an entropy-based perspective, proving that our method encourages diversity in the generated samples.