Efficient Financial Language Understanding via Distillation with Synthetic Data
For financial NLP practitioners with limited labeled data, this provides a practical, resource-efficient domain adaptation method.
The paper tackles financial sentiment analysis under low-resource conditions by distilling knowledge from a large teacher model into compact student models using synthetic data generated via clustering-based seed selection. The compact model outperforms the teacher on noisy text while remaining competitive on formal text.
Large instruction-following models are powerful but costly to deploy, particularly in finance, where labelled data are limited by confidentiality and expert annotation cost. We present an efficient framework for financial sentiment analysis through distillation with synthetic data, transferring knowledge from a large instruction-tuned teacher to compact student models. The framework is designed for low-resource conditions, where a small set of real examples are collected and labelled by hand. The framework then clusters the examples and uses the clusters to select seeds for generating synthetic examples via structured few-shot prompting. Experiments show that clustering-based seed selection yields more representative synthetic data than random sampling, enabling compact models to achieve strong performance with minimal supervision. Notably, on a more complex and noisy text domain, the compact model trained on the complete synthetic-seed corpus even outperforms the teacher model, while remaining competitive on formal text. The framework provides a practical route toward resource-efficient domain adaptation in financial NLP with minimal human labelling effort.