Simple, Distributed, and Accelerated Probabilistic Programming
This work provides a scalable and efficient framework for probabilistic programming, benefiting researchers and practitioners in machine learning by enabling faster and distributed computations.
The paper tackles the challenge of embedding probabilistic programming in deep learning ecosystems by introducing a simple abstraction based on random variables, achieving optimal linear speedup from 1 to 256 TPUv2 chips for models like VAE and Image Transformer, and a 100x speedup on GPUs for NUTS over Stan.
We describe a simple, low-level approach for embedding probabilistic programming in a deep learning ecosystem. In particular, we distill probabilistic programming down to a single abstraction---the random variable. Our lightweight implementation in TensorFlow enables numerous applications: a model-parallel variational auto-encoder (VAE) with 2nd-generation tensor processing units (TPUv2s); a data-parallel autoregressive model (Image Transformer) with TPUv2s; and multi-GPU No-U-Turn Sampler (NUTS). For both a state-of-the-art VAE on 64x64 ImageNet and Image Transformer on 256x256 CelebA-HQ, our approach achieves an optimal linear speedup from 1 to 256 TPUv2 chips. With NUTS, we see a 100x speedup on GPUs over Stan and 37x over PyMC3.