End-to-end learning of energy-based representations for irregularly-sampled signals and images
This addresses a critical issue for domains like earth observation and medical imaging where irregular sampling hinders the application of state-of-the-art learning methods, though it appears incremental as it builds on existing energy-based and neural network approaches.
The paper tackles the problem of learning representations from irregularly-sampled data with missing values, proposing energy-based methods derived from auto-encoders and Gibbs priors, and demonstrates their effectiveness on multivariate time series, 2D images, and image sequences.
For numerous domains, including for instance earth observation, medical imaging, astrophysics,..., available image and signal datasets often involve irregular space-time sampling patterns and large missing data rates. These sampling properties may be critical to apply state-of-the-art learning-based (e.g., auto-encoders, CNNs,...), fully benefit from the available large-scale observations and reach breakthroughs in the reconstruction and identification of processes of interest. In this paper, we address the end-to-end learning of representations of signals, images and image sequences from irregularly-sampled data, i.e. when the training data involved missing data. From an analogy to Bayesian formulation, we consider energy-based representations. Two energy forms are investigated: one derived from auto-encoders and one relating to Gibbs priors. The learning stage of these energy-based representations (or priors) involve a joint interpolation issue, which amounts to solving an energy minimization problem under observation constraints. Using a neural-network-based implementation of the considered energy forms, we can state an end-to-end learning scheme from irregularly-sampled data. We demonstrate the relevance of the proposed representations for different case-studies: namely, multivariate time series, 2D images and image sequences.