Convolutional dictionary learning based auto-encoders for natural exponential-family distributions
This work addresses the problem of handling non-Gaussian data like binomial and Poisson distributions for researchers in machine learning and data analysis, offering a scalable framework, but it is incremental as it combines existing ideas from convolutional generative models and deep learning.
The paper tackles modeling data from natural exponential-family distributions, such as count data, by introducing auto-encoder neural networks inspired by convolutional dictionary learning, resulting in better goodness-of-fit for neural spiking data and competitive performance for Poisson image denoising with fewer parameters.
We introduce a class of auto-encoder neural networks tailored to data from the natural exponential family (e.g., count data). The architectures are inspired by the problem of learning the filters in a convolutional generative model with sparsity constraints, often referred to as convolutional dictionary learning (CDL). Our work is the first to combine ideas from convolutional generative models and deep learning for data that are naturally modeled with a non-Gaussian distribution (e.g., binomial and Poisson). This perspective provides us with a scalable and flexible framework that can be re-purposed for a wide range of tasks and assumptions on the generative model. Specifically, the iterative optimization procedure for solving CDL, an unsupervised task, is mapped to an unfolded and constrained neural network, with iterative adjustments to the inputs to account for the generative distribution. We also show that the framework can easily be extended for discriminative training, appropriate for a supervised task. We demonstrate 1) that fitting the generative model to learn, in an unsupervised fashion, the latent stimulus that underlies neural spiking data leads to better goodness-of-fit compared to other baselines, 2) competitive performance compared to state-of-the-art algorithms for supervised Poisson image denoising, with significantly fewer parameters, and 3) gradient dynamics of shallow binomial auto-encoder.