Illusion3D: 3D Multiview Illusion with 2D Diffusion Priors
This addresses the challenge of creating more expressive and versatile 3D illusions for artists and designers, though it is incremental as it builds on existing diffusion methods.
The paper tackled the problem of automatically generating 3D multiview illusions from text or image inputs, achieving this by optimizing neural 3D representations using a pre-trained diffusion model to produce distinct interpretations from different viewing angles.
Automatically generating multiview illusions is a compelling challenge, where a single piece of visual content offers distinct interpretations from different viewing perspectives. Traditional methods, such as shadow art and wire art, create interesting 3D illusions but are limited to simple visual outputs (i.e., figure-ground or line drawing), restricting their artistic expressiveness and practical versatility. Recent diffusion-based illusion generation methods can generate more intricate designs but are confined to 2D images. In this work, we present a simple yet effective approach for creating 3D multiview illusions based on user-provided text prompts or images. Our method leverages a pre-trained text-to-image diffusion model to optimize the textures and geometry of neural 3D representations through differentiable rendering. When viewed from multiple angles, this produces different interpretations. We develop several techniques to improve the quality of the generated 3D multiview illusions. We demonstrate the effectiveness of our approach through extensive experiments and showcase illusion generation with diverse 3D forms.