On Contrastive Representations of Stochastic Processes
This work addresses a challenge in machine learning for applications like meta-learning and time series, offering a more robust alternative to traditional methods.
The authors tackled the problem of learning representations of stochastic processes, which often fails with high-dimensional or noisy observations, by proposing a contrastive framework (CReSP) that avoids exact reconstruction. They demonstrated its effectiveness on periodic functions, 3D objects, and dynamical processes, showing better tolerance to noise and improved transfer to downstream tasks.
Learning representations of stochastic processes is an emerging problem in machine learning with applications from meta-learning to physical object models to time series. Typical methods rely on exact reconstruction of observations, but this approach breaks down as observations become high-dimensional or noise distributions become complex. To address this, we propose a unifying framework for learning contrastive representations of stochastic processes (CReSP) that does away with exact reconstruction. We dissect potential use cases for stochastic process representations, and propose methods that accommodate each. Empirically, we show that our methods are effective for learning representations of periodic functions, 3D objects and dynamical processes. Our methods tolerate noisy high-dimensional observations better than traditional approaches, and the learned representations transfer to a range of downstream tasks.