FuseGen: PLM Fusion for Data-generation based Zero-shot Learning
This work addresses distribution bias in zero-shot learning for NLP practitioners, offering an incremental improvement over single-PLM methods.
The paper tackles the problem of low-quality synthetic datasets in data generation-based zero-shot learning by proposing FuseGen, a framework that uses multiple PLMs and trained STMs for iterative data generation and subset selection, resulting in substantial performance improvements across diverse tasks.
Data generation-based zero-shot learning, although effective in training Small Task-specific Models (STMs) via synthetic datasets generated by Pre-trained Language Models (PLMs), is often limited by the low quality of such synthetic datasets. Previous solutions have primarily focused on single PLM settings, where synthetic datasets are typically restricted to specific sub-spaces and often deviate from real-world distributions, leading to severe distribution bias. To mitigate such bias, we propose FuseGen, a novel data generation-based zero-shot learning framework that introduces a new criteria for subset selection from synthetic datasets via utilizing multiple PLMs and trained STMs. The chosen subset provides in-context feedback to each PLM, enhancing dataset quality through iterative data generation. Trained STMs are then used for sample re-weighting as well, further improving data quality. Extensive experiments across diverse tasks demonstrate that FuseGen substantially outperforms existing methods, highly effective in boosting STM performance in a PLM-agnostic way. Code is provided in https://github.com/LindaLydia/FuseGen.