Bit-Swap: Recursive Bits-Back Coding for Lossless Compression with Hierarchical Latent Variables
This addresses the open problem of practical bits-back coding for hierarchical models, offering incremental improvements in compression efficiency for data compression applications.
The paper tackles the problem of inefficient lossless compression for hierarchical latent variable models by proposing Bit-Swap, a scheme that generalizes BB-ANS and achieves strictly better compression rates, with experiments showing empirically superior performance.
The bits-back argument suggests that latent variable models can be turned into lossless compression schemes. Translating the bits-back argument into efficient and practical lossless compression schemes for general latent variable models, however, is still an open problem. Bits-Back with Asymmetric Numeral Systems (BB-ANS), recently proposed by Townsend et al. (2019), makes bits-back coding practically feasible for latent variable models with one latent layer, but it is inefficient for hierarchical latent variable models. In this paper we propose Bit-Swap, a new compression scheme that generalizes BB-ANS and achieves strictly better compression rates for hierarchical latent variable models with Markov chain structure. Through experiments we verify that Bit-Swap results in lossless compression rates that are empirically superior to existing techniques. Our implementation is available at https://github.com/fhkingma/bitswap.