M3DDM+: An improved video outpainting by a modified masking strategy
This work addresses video outpainting quality issues for computer vision applications, but it is incremental as it builds directly on the existing M3DDM framework.
The paper tackled the problem of quality degradation in video outpainting under challenging scenarios like limited camera motion or large outpainting regions, by modifying the masking strategy to align training and inference, resulting in substantial improvements in visual fidelity and temporal coherence while maintaining computational efficiency.
M3DDM provides a computationally efficient framework for video outpainting via latent diffusion modeling. However, it exhibits significant quality degradation -- manifested as spatial blur and temporal inconsistency -- under challenging scenarios characterized by limited camera motion or large outpainting regions, where inter-frame information is limited. We identify the cause as a training-inference mismatch in the masking strategy: M3DDM's training applies random mask directions and widths across frames, whereas inference requires consistent directional outpainting throughout the video. To address this, we propose M3DDM+, which applies uniform mask direction and width across all frames during training, followed by fine-tuning of the pretrained M3DDM model. Experiments demonstrate that M3DDM+ substantially improves visual fidelity and temporal coherence in information-limited scenarios while maintaining computational efficiency. The code is available at https://github.com/tamaki-lab/M3DDM-Plus.