PRISM: A Unified Framework for Photorealistic Reconstruction and Intrinsic Scene Modeling
This work addresses the need for efficient multi-task image generation and editing in computer vision, though it is incremental as it builds on existing diffusion models.
The paper tackles the problem of generating and editing images with intrinsic scene properties by proposing PRISM, a unified framework that fine-tunes a pre-trained diffusion model to produce RGB images and intrinsic maps simultaneously, achieving competitive performance in decomposition and conditional generation tasks.
We present PRISM, a unified framework that enables multiple image generation and editing tasks in a single foundational model. Starting from a pre-trained text-to-image diffusion model, PRISM proposes an effective fine-tuning strategy to produce RGB images along with intrinsic maps (referred to as X layers) simultaneously. Unlike previous approaches, which infer intrinsic properties individually or require separate models for decomposition and conditional generation, PRISM maintains consistency across modalities by generating all intrinsic layers jointly. It supports diverse tasks, including text-to-RGBX generation, RGB-to-X decomposition, and X-to-RGBX conditional generation. Additionally, PRISM enables both global and local image editing through conditioning on selected intrinsic layers and text prompts. Extensive experiments demonstrate the competitive performance of PRISM both for intrinsic image decomposition and conditional image generation while preserving the base model's text-to-image generation capability.