Variational Inference In Pachinko Allocation Machines
This addresses the problem of slow and limited inference in topic modeling for researchers, though it is incremental as it builds on existing variational autoencoder techniques.
The paper tackles the challenge of approximate inference in Pachinko Allocation Machines (PAM), a deep topic model, by introducing an efficient amortized variational inference method using a deep inference network. The result is more coherent topics than state-of-the-art methods and an order of magnitude faster inference, enabling exploration of a wider range of PAM architectures.
The Pachinko Allocation Machine (PAM) is a deep topic model that allows representing rich correlation structures among topics by a directed acyclic graph over topics. Because of the flexibility of the model, however, approximate inference is very difficult. Perhaps for this reason, only a small number of potential PAM architectures have been explored in the literature. In this paper we present an efficient and flexible amortized variational inference method for PAM, using a deep inference network to parameterize the approximate posterior distribution in a manner similar to the variational autoencoder. Our inference method produces more coherent topics than state-of-art inference methods for PAM while being an order of magnitude faster, which allows exploration of a wider range of PAM architectures than have previously been studied.