The Poisson Gamma Belief Network
This work addresses representation learning for count data in domains like text analysis, but it is incremental as it builds upon existing Poisson factor analysis methods.
The authors tackled the problem of inferring multilayer representations for high-dimensional count vectors by proposing the Poisson gamma belief network (PGBN), which factorizes layers and uses Gibbs sampling for training. Results on text analysis showed that PGBN outperforms Poisson factor analysis by adding more layers under the same constraints, with performance gains demonstrated.
To infer a multilayer representation of high-dimensional count vectors, we propose the Poisson gamma belief network (PGBN) that factorizes each of its layers into the product of a connection weight matrix and the nonnegative real hidden units of the next layer. The PGBN's hidden layers are jointly trained with an upward-downward Gibbs sampler, each iteration of which upward samples Dirichlet distributed connection weight vectors starting from the first layer (bottom data layer), and then downward samples gamma distributed hidden units starting from the top hidden layer. The gamma-negative binomial process combined with a layer-wise training strategy allows the PGBN to infer the width of each layer given a fixed budget on the width of the first layer. The PGBN with a single hidden layer reduces to Poisson factor analysis. Example results on text analysis illustrate interesting relationships between the width of the first layer and the inferred network structure, and demonstrate that the PGBN, whose hidden units are imposed with correlated gamma priors, can add more layers to increase its performance gains over Poisson factor analysis, given the same limit on the width of the first layer.