Embodied Self-supervised Learning by Coordinated Sampling and Training
This work addresses the challenge of making learned representations interpretable in physical terms for inverse problems, particularly in speech processing, offering a novel method that is incremental in its application to a specific domain.
The paper tackles the problem of learning representations with explicit physical meanings in self-supervised learning by proposing an analysis-by-synthesis approach that iteratively samples and trains using a physical forward process, applied to acoustic-to-articulatory inversion to infer articulatory information from speech, resulting in models that converge steadily, control an articulatory synthesizer to speak like a human, and generalize well to unseen speakers or new languages with further improvement through self-adaptation.
Self-supervised learning can significantly improve the performance of downstream tasks, however, the dimensions of learned representations normally lack explicit physical meanings. In this work, we propose a novel self-supervised approach to solve inverse problems by employing the corresponding physical forward process so that the learned representations can have explicit physical meanings. The proposed approach works in an analysis-by-synthesis manner to learn an inference network by iteratively sampling and training. At the sampling step, given observed data, the inference network is used to approximate the intractable posterior, from which we sample input parameters and feed them to a physical process to generate data in the observational space; At the training step, the same network is optimized with the sampled paired data. We prove the feasibility of the proposed method by tackling the acoustic-to-articulatory inversion problem to infer articulatory information from speech. Given an articulatory synthesizer, an inference model can be trained completely from scratch with random initialization. Our experiments demonstrate that the proposed method can converge steadily and the network learns to control the articulatory synthesizer to speak like a human. We also demonstrate that trained models can generalize well to unseen speakers or even new languages, and performance can be further improved through self-adaptation.