Big Data Scaling through Metric Mapping: Exploiting the Remarkable Simplicity of Very High Dimensional Spaces using Correspondence Analysis
This work addresses the problem of scaling data analysis in high-dimensional settings for researchers and practitioners in fields like chemistry and finance, but it appears incremental as it applies an existing method to new data without claiming major breakthroughs.
The paper tackles the challenge of analyzing data in very high-dimensional spaces (up to one million dimensions) by using Correspondence Analysis for orthonormal mapping, particularly suited for power law distributed data common in domains like digital chemistry and finance, with experiments conducted on these and statistically generated datasets.
We present new findings in regard to data analysis in very high dimensional spaces. We use dimensionalities up to around one million. A particular benefit of Correspondence Analysis is its suitability for carrying out an orthonormal mapping, or scaling, of power law distributed data. Power law distributed data are found in many domains. Correspondence factor analysis provides a latent semantic or principal axes mapping. Our experiments use data from digital chemistry and finance, and other statistically generated data.