Class Density and Dataset Quality in High-Dimensional, Unstructured Data
This work addresses dataset quality assessment for high-dimensional unstructured data, which is incremental as it builds on existing methods for similarity measurement.
The paper tackled the problem of measuring class density and dataset quality in high-dimensional unstructured data, showing that datasets meeting a quality threshold (>10) could be candidates for eliding redundant data based on class densities.
We provide a definition for class density that can be used to measure the aggregate similarity of the samples within each of the classes in a high-dimensional, unstructured dataset. We then put forth several candidate methods for calculating class density and analyze the correlation between the values each method produces with the corresponding individual class test accuracies achieved on a trained model. Additionally, we propose a definition for dataset quality for high-dimensional, unstructured data and show that those datasets that met a certain quality threshold (experimentally demonstrated to be > 10 for the datasets studied) were candidates for eliding redundant data based on the individual class densities.