SceneSeer: 3D Scene Design with Natural Language
This addresses the challenge of making 3D scene design more accessible to non-experts by reducing the need for complex interfaces, though it is incremental as it builds on existing knowledge bases and methods.
The paper tackles the problem of designing 3D scenes by introducing SceneSeer, an interactive system that generates 3D scenes from natural language input, allowing iterative refinement through textual commands; it shows that the generated scenes are comparable to manually designed ones in perceptual evaluations.
Designing 3D scenes is currently a creative task that requires significant expertise and effort in using complex 3D design interfaces. This effortful design process starts in stark contrast to the easiness with which people can use language to describe real and imaginary environments. We present SceneSeer: an interactive text to 3D scene generation system that allows a user to design 3D scenes using natural language. A user provides input text from which we extract explicit constraints on the objects that should appear in the scene. Given these explicit constraints, the system then uses a spatial knowledge base learned from an existing database of 3D scenes and 3D object models to infer an arrangement of the objects forming a natural scene matching the input description. Using textual commands the user can then iteratively refine the created scene by adding, removing, replacing, and manipulating objects. We evaluate the quality of 3D scenes generated by SceneSeer in a perceptual evaluation experiment where we compare against manually designed scenes and simpler baselines for 3D scene generation. We demonstrate how the generated scenes can be iteratively refined through simple natural language commands.