Building a Pilot Software Quality-in-Use Benchmark Dataset
This provides a benchmark dataset for evaluating sentiment analysis models in software quality-in-use, but it is incremental as it focuses on data creation rather than novel methods.
The authors tackled the lack of domain-specific datasets for software quality-in-use by creating a new annotated sentence dataset, resulting in moderate to substantial expert agreement (Kappa statistics) and a reusable annotation scheme.
Prepared domain specific datasets plays an important role to supervised learning approaches. In this article a new sentence dataset for software quality-in-use is proposed. Three experts were chosen to annotate the data using a proposed annotation scheme. Then the data were reconciled in a (no match eliminate) process to reduce bias. The Kappa, k statistics revealed an acceptable level of agreement; moderate to substantial agreement between the experts. The built data can be used to evaluate software quality-in-use models in sentiment analysis models. Moreover, the annotation scheme can be used to extend the current dataset.