Unsupervised Feature Extraction by Time-Contrastive Learning and Nonlinear ICA
This work addresses a foundational issue in unsupervised feature learning for machine learning and AI, offering a novel solution to the long-standing identifiability problem in nonlinear ICA.
The paper tackles the identifiability problem in nonlinear independent component analysis (ICA) by introducing time-contrastive learning (TCL), an unsupervised deep learning principle that leverages temporal nonstationarities in time series data. It shows that TCL combined with linear ICA uniquely estimates the nonlinear ICA model up to point-wise transformations, providing the first rigorous and general identifiability result.
Nonlinear independent component analysis (ICA) provides an appealing framework for unsupervised feature learning, but the models proposed so far are not identifiable. Here, we first propose a new intuitive principle of unsupervised deep learning from time series which uses the nonstationary structure of the data. Our learning principle, time-contrastive learning (TCL), finds a representation which allows optimal discrimination of time segments (windows). Surprisingly, we show how TCL can be related to a nonlinear ICA model, when ICA is redefined to include temporal nonstationarities. In particular, we show that TCL combined with linear ICA estimates the nonlinear ICA model up to point-wise transformations of the sources, and this solution is unique --- thus providing the first identifiability result for nonlinear ICA which is rigorous, constructive, as well as very general.