Dimensionality reduction to maximize prediction generalization capability
This addresses generalization issues in time series prediction for machine learning applications, with potential neuromorphic hardware benefits due to its simple architecture and low computational cost.
The paper tackles the problem of generalization in time series prediction by developing PredPCA, an unsupervised dimensionality reduction method that extracts informative components for predicting future inputs. The method asymptotically identifies hidden states and system parameters with global convergence guarantees, and demonstrates reliable performance on sequential visual inputs including hand-digits, rotating 3D objects, and natural scenes.
Generalization of time series prediction remains an important open issue in machine learning, wherein earlier methods have either large generalization error or local minima. We develop an analytically solvable, unsupervised learning scheme that extracts the most informative components for predicting future inputs, termed predictive principal component analysis (PredPCA). Our scheme can effectively remove unpredictable noise and minimize test prediction error through convex optimization. Mathematical analyses demonstrate that, provided with sufficient training samples and sufficiently high-dimensional observations, PredPCA can asymptotically identify hidden states, system parameters, and dimensionalities of canonical nonlinear generative processes, with a global convergence guarantee. We demonstrate the performance of PredPCA using sequential visual inputs comprising hand-digits, rotating 3D objects, and natural scenes. It reliably estimates distinct hidden states and predicts future outcomes of previously unseen test input data, based exclusively on noisy observations. The simple architecture and low computational cost of PredPCA are highly desirable for neuromorphic hardware.