Improving Unsupervised Visual Program Inference with Code Rewriting Families
This work addresses visual program inference for domains like 2D and 3D shape programming, offering incremental improvements through code rewriting techniques.
The paper tackles the problem of inferring programs from visual data by proposing Sparse Intermittent Rewrite Injection (SIRI) with a family of code rewriters, showing improved reconstructions and faster convergence rates compared to baseline methods, and matching or outperforming recent domain-specific neural architectures while using significantly fewer primitives.
Programs offer compactness and structure that makes them an attractive representation for visual data. We explore how code rewriting can be used to improve systems for inferring programs from visual data. We first propose Sparse Intermittent Rewrite Injection (SIRI), a framework for unsupervised bootstrapped learning. SIRI sparsely applies code rewrite operations over a dataset of training programs, injecting the improved programs back into the training set. We design a family of rewriters for visual programming domains: parameter optimization, code pruning, and code grafting. For three shape programming languages in 2D and 3D, we show that using SIRI with our family of rewriters improves performance: better reconstructions and faster convergence rates, compared with bootstrapped learning methods that do not use rewriters or use them naively. Finally, we demonstrate that our family of rewriters can be effectively used at test time to improve the output of SIRI predictions. For 2D and 3D CSG, we outperform or match the reconstruction performance of recent domain-specific neural architectures, while producing more parsimonious programs that use significantly fewer primitives.