Go with the Flow: Perception-refined Physics Simulation
This addresses the challenge of extracting physical parameters from visual data for applications in computer vision and physics-based modeling, though it appears incremental as it builds on existing simulation and embedding techniques.
The paper tackles the problem of inferring latent physical properties from visual observations by proposing an iterative refinement procedure that compares simulations to real-world observations using a physically-aware embedding function. They demonstrate the method's ability to recover physical parameters from simulated and real-world video of flags curling in the wind.
For many of the physical phenomena around us, we have developed sophisticated models explaining their behavior. Nevertheless, inferring specifics from visual observations is challenging due to the high number of causally underlying physical parameters -- including material properties and external forces. This paper addresses the problem of inferring such latent physical properties from observations. Our solution is an iterative refinement procedure with simulation at its core. The algorithm gradually updates the physical model parameters by running a simulation of the observed phenomenon and comparing the current simulation to a real-world observation. The physical similarity is computed using an embedding function that maps physically similar examples to nearby points. As a tangible example, we concentrate on flags curling in the wind -- a seemingly simple phenomenon but physically highly involved. Based on its underlying physical model and visual manifestation, we propose an instantiation of the embedding function. For this mapping, modeled as a deep network, we introduce a spectral decomposition layer that decomposes a video volume into its temporal spectral power and corresponding frequencies. In experiments, we demonstrate our method's ability to recover intrinsic and extrinsic physical parameters from both simulated and real-world video.