Dynamic Texture Synthesis by Incorporating Long-range Spatial and Temporal Correlations
This work improves dynamic texture synthesis for applications like video generation, though it is incremental as it extends existing models with new techniques.
The paper tackled the challenge of maintaining spatial and temporal consistency in dynamic texture synthesis by addressing poor treatment of long-range correlations and motion, resulting in a model that synthesizes both homogeneous and structured patterns with state-of-the-art visual performance.
The main challenge of dynamic texture synthesis lies in how to maintain spatial and temporal consistency in synthesized videos. The major drawback of existing dynamic texture synthesis models comes from poor treatment of the long-range texture correlation and motion information. To address this problem, we incorporate a new loss term, called the Shifted Gram loss, to capture the structural and long-range correlation of the reference texture video. Furthermore, we introduce a frame sampling strategy to exploit long-period motion across multiple frames. With these two new techniques, the application scope of existing texture synthesis models can be extended. That is, they can synthesize not only homogeneous but also structured dynamic texture patterns. Thorough experimental results are provided to demonstrate that our proposed dynamic texture synthesis model offers state-of-the-art visual performance.