Desiderata for Representation Learning: A Causal Perspective
This work addresses the problem of making representation learning criteria measurable and actionable for researchers and practitioners, though it is incremental in formalizing existing concepts.
The paper tackles the challenge of formalizing intuitive desiderata like non-spuriousness, efficiency, and disentanglement in representation learning by adopting a causal perspective, resulting in computable metrics for assessing and learning such representations from observational data.
Representation learning constructs low-dimensional representations to summarize essential features of high-dimensional data. This learning problem is often approached by describing various desiderata associated with learned representations; e.g., that they be non-spurious, efficient, or disentangled. It can be challenging, however, to turn these intuitive desiderata into formal criteria that can be measured and enhanced based on observed data. In this paper, we take a causal perspective on representation learning, formalizing non-spuriousness and efficiency (in supervised representation learning) and disentanglement (in unsupervised representation learning) using counterfactual quantities and observable consequences of causal assertions. This yields computable metrics that can be used to assess the degree to which representations satisfy the desiderata of interest and learn non-spurious and disentangled representations from single observational datasets.