Learning Neurosymbolic Generative Models via Program Synthesis
This addresses a key limitation in generative modeling for tasks like image generation, though it appears incremental as it builds on existing methods by adding programmatic components.
The paper tackles the problem of generative models failing to capture complex global structure in data, such as repeating patterns in images, by incorporating programs representing that structure and using program synthesis for learning. The result is a framework that substantially outperforms state-of-the-art methods in generating and completing images with global structure on synthetic and real-world data.
Significant strides have been made toward designing better generative models in recent years. Despite this progress, however, state-of-the-art approaches are still largely unable to capture complex global structure in data. For example, images of buildings typically contain spatial patterns such as windows repeating at regular intervals; state-of-the-art generative methods can't easily reproduce these structures. We propose to address this problem by incorporating programs representing global structure into the generative model---e.g., a 2D for-loop may represent a configuration of windows. Furthermore, we propose a framework for learning these models by leveraging program synthesis to generate training data. On both synthetic and real-world data, we demonstrate that our approach is substantially better than the state-of-the-art at both generating and completing images that contain global structure.