Learning to Compile Programs to Neural Networks
This addresses the trade-off between resource consumption and accuracy in neural surrogate generation for tasks like program tuning and acceleration, offering a more efficient approach.
The paper tackles the problem of creating neural surrogates for programs by introducing neural surrogate compilation, which directly produces neural networks from program text without coupling generation and execution, resulting in 1.9-9.5x data efficiency, 1.0-1.3x similarity to ground truth, and 4.3-7.3x faster training compared to traditional methods.
A $\textit{neural surrogate of a program}$ is a neural network that mimics the behavior of a program. Researchers have used these neural surrogates to automatically tune program inputs, adapt programs to new settings, and accelerate computations. Researchers traditionally develop neural surrogates by training on input-output examples from a single program. Alternatively, language models trained on a large dataset including many programs can consume program text, to act as a neural surrogate. Using a language model to both generate a surrogate and act as a surrogate, however, leading to a trade-off between resource consumption and accuracy. We present $\textit{neural surrogate compilation}$, a technique for producing neural surrogates directly from program text without coupling neural surrogate generation and execution. We implement neural surrogate compilers using hypernetworks trained on a dataset of C programs and find that they produce neural surrogates that are $1.9$-$9.5\times$ as data-efficient, produce visual results that are $1.0$-$1.3\times$ more similar to ground truth, and train in $4.3$-$7.3\times$ fewer epochs than neural surrogates trained from scratch.