Evaluation and Improvement of Chatbot Text Classification Data Quality Using Plausible Negative Examples
This work addresses the need for understandable and useful evaluation metrics for chatbot datasets in recruitment, though it is incremental as it adapts existing cross-validation methods.
The authors tackled the problem of evaluating and improving chatbot text classification data quality by introducing a metric called nex-cv that uses plausible negative examples, validated on seven recruitment-domain datasets over one year, showing it is actionable, not overly optimistic, and model-agnostic.
We describe and validate a metric for estimating multi-class classifier performance based on cross-validation and adapted for improvement of small, unbalanced natural-language datasets used in chatbot design. Our experiences draw upon building recruitment chatbots that mediate communication between job-seekers and recruiters by exposing the ML/NLP dataset to the recruiting team. Evaluation approaches must be understandable to various stakeholders, and useful for improving chatbot performance. The metric, nex-cv, uses negative examples in the evaluation of text classification, and fulfils three requirements. First, it is actionable: it can be used by non-developer staff. Second, it is not overly optimistic compared to human ratings, making it a fast method for comparing classifiers. Third, it allows model-agnostic comparison, making it useful for comparing systems despite implementation differences. We validate the metric based on seven recruitment-domain datasets in English and German over the course of one year.