LogitBoost autoregressive networks
This work addresses the challenge of efficient and scalable modeling for multivariate binary data, though it is incremental as it builds on existing decomposition methods.
The paper tackled the problem of modeling multivariate binary distributions by showing that training separate probability estimators for each dimension can achieve state-of-the-art performance on standard benchmarks, with model training being trivially parallelizable over data dimensions.
Multivariate binary distributions can be decomposed into products of univariate conditional distributions. Recently popular approaches have modeled these conditionals through neural networks with sophisticated weight-sharing structures. It is shown that state-of-the-art performance on several standard benchmark datasets can actually be achieved by training separate probability estimators for each dimension. In that case, model training can be trivially parallelized over data dimensions. On the other hand, complexity control has to be performed for each learned conditional distribution. Three possible methods are considered and experimentally compared. The estimator that is employed for each conditional is LogitBoost. Similarities and differences between the proposed approach and autoregressive models based on neural networks are discussed in detail.