Sketchforme: Composing Sketched Scenes from Text Descriptions for Interactive Applications
This work addresses the need for interactive applications like language learning and sketching assistants by enabling text-to-sketch generation.
The authors tackled the problem of generating sketches from text descriptions without paired training data, resulting in sketches that were judged to convey descriptions better than human sketches in some cases and were considered human-generated 36.5% of the time.
Sketching and natural languages are effective communication media for interactive applications. We introduce Sketchforme, the first neural-network-based system that can generate sketches based on text descriptions specified by users. Sketchforme is capable of gaining high-level and low-level understanding of multi-object sketched scenes without being trained on sketched scene datasets annotated with text descriptions. The sketches composed by Sketchforme are expressive and realistic: we show in our user study that these sketches convey descriptions better than human-generated sketches in multiple cases, and 36.5% of those sketches are considered to be human-generated. We develop multiple interactive applications using these generated sketches, and show that Sketchforme can significantly improve language learning applications and support intelligent language-based sketching assistants.