Compressed Predictive Information Coding
This work addresses the lag in methods for dynamic data representation extraction, which is important for fields like AI, machine learning, and neuroscience, but it appears incremental as it builds on existing information-theoretic and variational approaches.
The authors tackled the problem of extracting low-dimensional representations from dynamic data by developing Compressed Predictive Information Coding (CPIC), a novel information-theoretic framework that learns representations by minimizing compression complexity and maximizing predictive information, and demonstrated its ability to recover latent spaces in noisy dynamical systems and extract predictive features in neuroscience data.
Unsupervised learning plays an important role in many fields, such as artificial intelligence, machine learning, and neuroscience. Compared to static data, methods for extracting low-dimensional structure for dynamic data are lagging. We developed a novel information-theoretic framework, Compressed Predictive Information Coding (CPIC), to extract useful representations from dynamic data. CPIC selectively projects the past (input) into a linear subspace that is predictive about the compressed data projected from the future (output). The key insight of our framework is to learn representations by minimizing the compression complexity and maximizing the predictive information in latent space. We derive variational bounds of the CPIC loss which induces the latent space to capture information that is maximally predictive. Our variational bounds are tractable by leveraging bounds of mutual information. We find that introducing stochasticity in the encoder robustly contributes to better representation. Furthermore, variational approaches perform better in mutual information estimation compared with estimates under a Gaussian assumption. We demonstrate that CPIC is able to recover the latent space of noisy dynamical systems with low signal-to-noise ratios, and extracts features predictive of exogenous variables in neuroscience data.