Improving the Accuracy of Principal Component Analysis by the Maximum Entropy Method
This work addresses a long-standing issue in data analysis by enhancing PCA's accuracy for tasks like approximate nearest neighbor search, though it is incremental as it builds on a century-old technique.
The paper tackles the problem of inaccuracies in distance estimation from Principal Component Analysis (PCA) approximations by modeling uncertainty with random variables and applying the Maximum Entropy Method to infer probability distributions, resulting in improved accuracy over classical PCA in most cases, as shown experimentally.
Classical Principal Component Analysis (PCA) approximates data in terms of projections on a small number of orthogonal vectors. There are simple procedures to efficiently compute various functions of the data from the PCA approximation. The most important function is arguably the Euclidean distance between data items, This can be used, for example, to solve the approximate nearest neighbor problem. We use random variables to model the inherent uncertainty in such approximations, and apply the Maximum Entropy Method to infer the underlying probability distribution. We propose using the expected values of distances between these random variables as improved estimates of the distance. We show by analysis and experimentally that in most cases results obtained by our method are more accurate than what is obtained by the classical approach. This improves the accuracy of a classical technique that have been used with little change for over 100 years.