Refining Filter Global Feature Weighting for Fully-Unsupervised Clustering
This work addresses the challenge of feature relevance in clustering for data scientists, though it is incremental as it applies an existing explainability technique to a new context.
The paper tackled the problem of improving unsupervised clustering by weighting features based on SHAP values, achieving up to a 22.69% improvement in clustering quality over other methods, as measured by the Adjusted Rand Index.
In the context of unsupervised learning, effective clustering plays a vital role in revealing patterns and insights from unlabeled data. However, the success of clustering algorithms often depends on the relevance and contribution of features, which can differ between various datasets. This paper explores feature weighting for clustering and presents new weighting strategies, including methods based on SHAP (SHapley Additive exPlanations), a technique commonly used for providing explainability in various supervised machine learning tasks. By taking advantage of SHAP values in a way other than just to gain explainability, we use them to weight features and ultimately improve the clustering process itself in unsupervised scenarios. Our empirical evaluations across five benchmark datasets and clustering methods demonstrate that feature weighting based on SHAP can enhance unsupervised clustering quality, achieving up to a 22.69\% improvement over other weighting methods (from 0.586 to 0.719 in terms of the Adjusted Rand Index). Additionally, these situations where the weighted data boosts the results are highlighted and thoroughly explored, offering insight for practical applications.