Optimizing Text Quantifiers for Multivariate Loss Functions
This work addresses the need for more accurate and stable quantification methods in data and text mining applications, such as analyzing reviews or support calls, representing an incremental improvement over existing approaches.
The paper tackles the problem of quantification, which estimates class prevalence in unlabeled datasets, by introducing a structured prediction model that directly optimizes for multivariate loss functions, resulting in improved accuracy, stability, and efficiency over state-of-the-art methods on 5500 binary high-dimensional datasets.
We address the problem of \emph{quantification}, a supervised learning task whose goal is, given a class, to estimate the relative frequency (or \emph{prevalence}) of the class in a dataset of unlabelled items. Quantification has several applications in data and text mining, such as estimating the prevalence of positive reviews in a set of reviews of a given product, or estimating the prevalence of a given support issue in a dataset of transcripts of phone calls to tech support. So far, quantification has been addressed by learning a general-purpose classifier, counting the unlabelled items which have been assigned the class, and tuning the obtained counts according to some heuristics. In this paper we depart from the tradition of using general-purpose classifiers, and use instead a supervised learning model for \emph{structured prediction}, capable of generating classifiers directly optimized for the (multivariate and non-linear) function used for evaluating quantification accuracy. The experiments that we have run on 5500 binary high-dimensional datasets (averaging more than 14,000 documents each) show that this method is more accurate, more stable, and more efficient than existing, state-of-the-art quantification methods.