Comprehensive and Efficient Distillation for Lightweight Sentiment Analysis Models
This work addresses the problem of creating efficient and practical lightweight sentiment analysis models for real-world applications, representing an incremental improvement over existing distillation methods.
The paper tackled the challenges of limited instruction diversity and high computational costs in knowledge distillation for sentiment analysis by introducing CompEffDist, a framework with automatic instruction construction and data filtering, enabling 3B student models to match 20x larger teacher models and achieve the same performance with only 10% of the data.
Recent efforts leverage knowledge distillation techniques to develop lightweight and practical sentiment analysis models. These methods are grounded in human-written instructions and large-scale user texts. Despite the promising results, two key challenges remain: (1) manually written instructions are limited in diversity and quantity, making them insufficient to ensure comprehensive coverage of distilled knowledge; (2) large-scale user texts incur high computational cost, hindering the practicality of these methods. To this end, we introduce CompEffDist, a comprehensive and efficient distillation framework for sentiment analysis. Our framework consists of two key modules: attribute-based automatic instruction construction and difficulty-based data filtering, which correspondingly tackle the aforementioned challenges. Applying our method across multiple model series (Llama-3, Qwen-3, and Gemma-3), we enable 3B student models to match the performance of 20x larger teacher models on most tasks. In addition, our approach greatly outperforms baseline methods in data efficiency, attaining the same performance level with only 10% of the data.