CinemaWorld: Generative Augmented Reality with LLMs and 3D Scene Generation for Movie Augmentation
This system aims to enrich the film-viewing experience for general audiences by enhancing immersion and enjoyment through augmented reality, representing an incremental step in AR entertainment.
CinemaWorld is a generative augmented reality system that augments physical surroundings with 3D content from 2D movie scenes. It uses multimodal LLMs to extract features from films and generative AI to create dynamic 3D augmentations, which are then embedded into the viewer's environment on the Meta Quest 3. The system was evaluated with 100 video clips, 12 users, and 8 film creators, showing enhanced immersion and enjoyment.
We introduce CinemaWorld, a generative augmented reality system that augments the viewer's physical surroundings with automatically generated mixed reality 3D content extracted from and synchronized with 2D movie scenes. Our system preprocesses films to extract key features using multimodal large language models (LLMs), generates dynamic 3D augmentations with generative AI, and embeds them spatially into the viewer's physical environment on the Meta Quest 3. To explore the design space of CinemaWorld, we conducted an elicitation study with eight film students, which led us to identify several key augmentation types, including particle effects, surrounding objects, textural overlays, character-driven augmentation, and lighting effects. We evaluated our system through a technical evaluation (N=100 video clips), a user study (N=12), and expert interviews with film creators (N=8). Results indicate that CinemaWorld enhances immersion and enjoyment, suggesting its potential to enrich the film-viewing experience.