Discovering Reliable Correlations in Categorical Data
This work addresses the need for interpretable and assumption-free correlation discovery in scientific tasks involving categorical data, representing an incremental improvement with specific algorithmic gains.
The paper tackles the problem of discovering reliable correlations in categorical data by proposing a non-parametric estimator for normalized total correlation and an algorithmic framework for efficient top-k set discovery, with empirical results showing low-regret outcomes for small sample sizes and effectiveness in large, high-dimensional datasets.
In many scientific tasks we are interested in discovering whether there exist any correlations in our data. This raises many questions, such as how to reliably and interpretably measure correlation between a multivariate set of attributes, how to do so without having to make assumptions on distribution of the data or the type of correlation, and, how to efficiently discover the top-most reliably correlated attribute sets from data. In this paper we answer these questions for discovery tasks in categorical data. In particular, we propose a corrected-for-chance, consistent, and efficient estimator for normalized total correlation, by which we obtain a reliable, naturally interpretable, non-parametric measure for correlation over multivariate sets. For the discovery of the top-k correlated sets, we derive an effective algorithmic framework based on a tight bounding function. This framework offers exact, approximate, and heuristic search. Empirical evaluation shows that already for small sample sizes the estimator leads to low-regret optimization outcomes, while the algorithms are shown to be highly effective for both large and high-dimensional data. Through two case studies we confirm that our discovery framework identifies interesting and meaningful correlations.