Hamiltonian latent operators for content and motion disentanglement in image sequences
This work addresses the challenge of reliably separating static and dynamic components in video data for applications in computer vision and graphics, representing an incremental advancement in disentanglement methods.
The paper tackles the problem of disentangling content and motion in image sequences by introducing HALO, a deep generative model that uses Hamiltonian latent operators to ensure reversible and continuous motion dynamics, resulting in improved long-term sequence generation and controlled tasks like motion swapping.
We introduce \textit{HALO} -- a deep generative model utilising HAmiltonian Latent Operators to reliably disentangle content and motion information in image sequences. The \textit{content} represents summary statistics of a sequence, and \textit{motion} is a dynamic process that determines how information is expressed in any part of the sequence. By modelling the dynamics as a Hamiltonian motion, important desiderata are ensured: (1) the motion is reversible, (2) the symplectic, volume-preserving structure in phase space means paths are continuous and are not divergent in the latent space. Consequently, the nearness of sequence frames is realised by the nearness of their coordinates in the phase space, which proves valuable for disentanglement and long-term sequence generation. The sequence space is generally comprised of different types of dynamical motions. To ensure long-term separability and allow controlled generation, we associate every motion with a unique Hamiltonian that acts in its respective subspace. We demonstrate the utility of \textit{HALO} by swapping the motion of a pair of sequences, controlled generation, and image rotations.