NanoNet: Parameter-Efficient Learning with Label-Scarce Supervision for Lightweight Text Mining Model
This work addresses the challenge of efficient model training for text mining tasks with limited labeled data, though it appears incremental as it builds on existing lightweight semi-supervised learning strategies.
The paper tackles the problem of computationally intensive and suboptimal training in lightweight semi-supervised learning for text mining by proposing NanoNet, a framework that integrates limited labeled supervision, lightweight fine-tuning, and rapid-inference small models, resulting in reduced training costs and minimized supervision requirements.
The lightweight semi-supervised learning (LSL) strategy provides an effective approach of conserving labeled samples and minimizing model inference costs. Prior research has effectively applied knowledge transfer learning and co-training regularization from large to small models in LSL. However, such training strategies are computationally intensive and prone to local optima, thereby increasing the difficulty of finding the optimal solution. This has prompted us to investigate the feasibility of integrating three low-cost scenarios for text mining tasks: limited labeled supervision, lightweight fine-tuning, and rapid-inference small models. We propose NanoNet, a novel framework for lightweight text mining that implements parameter-efficient learning with limited supervision. It employs online knowledge distillation to generate multiple small models and enhances their performance through mutual learning regularization. The entire process leverages parameter-efficient learning, reducing training costs and minimizing supervision requirements, ultimately yielding a lightweight model for downstream inference.