Differentiable Random Partition Models
This addresses a fundamental limitation in machine learning for tasks like clustering and multitask learning, offering a general-purpose solution, though it appears incremental as it builds on existing random partition models.
The paper tackles the problem of partitioning elements into an unknown number of subsets, which is non-differentiable and hinders gradient-based optimization, by proposing a novel two-step method that enables reparameterized gradients for variational inference tasks. The result is demonstrated through experiments in variational clustering, generative factor inference, and multitask learning.
Partitioning a set of elements into an unknown number of mutually exclusive subsets is essential in many machine learning problems. However, assigning elements, such as samples in a dataset or neurons in a network layer, to an unknown and discrete number of subsets is inherently non-differentiable, prohibiting end-to-end gradient-based optimization of parameters. We overcome this limitation by proposing a novel two-step method for inferring partitions, which allows its usage in variational inference tasks. This new approach enables reparameterized gradients with respect to the parameters of the new random partition model. Our method works by inferring the number of elements per subset and, second, by filling these subsets in a learned order. We highlight the versatility of our general-purpose approach on three different challenging experiments: variational clustering, inference of shared and independent generative factors under weak supervision, and multitask learning.