Why Latent Actions Fail, and How to Prevent It
Provides a unified theoretical analysis of a fundamental problem in unsupervised action learning from videos, benefiting researchers working on video understanding and representation learning.
The paper identifies that exogenous state (e.g., background clutter) in videos hinders latent action learning by introducing irrelevant changes, and shows that minimizing reconstruction loss causes latent actions to encode exogenous information. It proves that learning in an endogenous-focused representation space and using auxiliary objectives like action-supervision mitigate this issue.
Latent action models (LAMs) aim to learn action-like representations from unlabeled videos by compressing frame-to-frame changes. The frames of in-the-wild videos, however, contain not only the agent's own state but exogenous state such as background clutter. Since the exogenous state introduces changes unrelated to actions, it hinders reliable latent action learning. This paper investigates this problem analytically by extending a linear LAM framework to explicitly model exogenous state. Our analysis reveals two insights: (1) minimizing the standard reconstruction objective produces latent actions that encode exogenous information from future observation; and (2) learning in a representation space that focuses on endogenous components is a key to mitigating the interference of noise. We further show that previously proposed auxiliary objectives, such as action-supervision, provably encourage latent actions to be consistent across exogenous states. These findings are validated through experiments on both linear and nonlinear LAMs, providing a unified theoretical analysis of how exogenous state hinders latent action learning and why common remedies work.