Vibe Coding XR: Accelerating AI + XR Prototyping with XR Blocks and Gemini
This work democratizes spatial software creation for practitioners by enabling rapid prototyping of XR applications, though it appears incremental as it builds on existing 'vibe coding' concepts.
The authors tackled the problem of inaccessible prototyping for intelligent Extended Reality (XR) experiences by developing XR Blocks, a modular WebXR framework, and Vibe Coding XR, a workflow that uses LLMs to translate natural language prompts into functional XR software in under a minute.
While large language models have accelerated software development through "vibe coding", prototyping intelligent Extended Reality (XR) experiences remains inaccessible due to the friction of complex game engines and low-level sensor integration. To bridge this gap, we contribute XR Blocks, an open-source, modular WebXR framework that abstracts spatial computing complexities into high-level, human-centered primitives. Building upon this foundation, we present Vibe Coding XR, an end-to-end rapid prototyping workflow that leverages LLMs to translate natural language intent directly into functional XR software. Using a web-based interface, creators can transform high-level prompts (e.g., "create a dandelion that reacts to hand") into interactive WebXR applications in under a minute. We provide a preliminary technical evaluation on a pilot dataset (VCXR60) alongside diverse application scenarios highlighting mixed-reality realism, multi-modal interaction, and generative AI integrations. By democratizing spatial software creation, this work empowers practitioners to bypass low-level hurdles and rapidly move from "idea to reality." Code and live demos are available at https://xrblocks.github.io/gem and https://github.com/google/xrblocks.