Learning to Predict Usage Options of Product Reviews with LLM-Generated Labels
This addresses data annotation challenges for NLP practitioners by providing a cost-effective alternative to crowd-sourcing and vendor services.
The authors tackled the problem of expensive and low-quality data annotation for complex NLP tasks by using LLMs as few-shot learners to generate labels for predicting product usage options from customer reviews. They demonstrated that GPT-4-generated labels reach domain expert quality and enable considerable cost savings compared to traditional methods.
Annotating large datasets can be challenging. However, crowd-sourcing is often expensive and can lack quality, especially for non-trivial tasks. We propose a method of using LLMs as few-shot learners for annotating data in a complex natural language task where we learn a standalone model to predict usage options for products from customer reviews. We also propose a new evaluation metric for this scenario, HAMS4, that can be used to compare a set of strings with multiple reference sets. Learning a custom model offers individual control over energy efficiency and privacy measures compared to using the LLM directly for the sequence-to-sequence task. We compare this data annotation approach with other traditional methods and demonstrate how LLMs can enable considerable cost savings. We find that the quality of the resulting data exceeds the level attained by third-party vendor services and that GPT-4-generated labels even reach the level of domain experts. We make the code and generated labels publicly available.