Amortizing Pragmatic Program Synthesis with Rankings
This work addresses efficiency bottlenecks in program synthesis for human-computer interaction, offering a practical improvement over existing methods.
The paper tackles the slow speed of pragmatic program synthesizers using the Rational Speech Acts framework by amortizing the exact synthesizer to compile a dataset and distill a global ranking for fast inference, achieving orders of magnitude speed-ups while maintaining higher accuracy than non-pragmatic synthesizers in human communication.
The usage of Rational Speech Acts (RSA) framework has been successful in building \emph{pragmatic} program synthesizers that return programs which, in addition to being logically consistent with user-generated examples, account for the fact that a user chooses their examples informatively. We present a general method of amortizing the slow, exact RSA synthesizer. Our method first query the exact RSA synthesizer to compile a communication dataset. The dataset contains a number of example-dependent rankings of subsets of programs. It then distills a \textit{single} global ranking of all programs as an approximation to every ranking in the dataset. This global ranking is then used at inference time to rank multiple logically consistent candidate programs generated from a fast, non-pragmatic synthesizer. Experiments on two program synthesis domains using our ranking method resulted in orders of magnitudes of speed ups compared to the exact RSA synthesizer, while being more accurate than a non-pragmatic synthesizer when communicating with humans. Finally, we prove that in the special case of synthesis from a single example, this approximation is exact.