Compact Compositional Models
This work addresses the challenge of representation learning for binary data, which is an incremental improvement in a domain-specific area.
The paper tackles the problem of learning compact and interpretable representations for binary data by proposing a new composition rule that discourages experts from focusing on similar structures and penalizes opposing votes strongly, along with a sequential initialization procedure; experiments show that this approach yields very intuitive models.
Learning compact and interpretable representations is a very natural task, which has not been solved satisfactorily even for simple binary datasets. In this paper, we review various ways of composing experts for binary data and argue that competitive forms of interaction are best suited to learn low-dimensional representations. We propose a new composition rule that discourages experts from focusing on similar structures and that penalizes opposing votes strongly so that abstaining from voting becomes more attractive. We also introduce a novel sequential initialization procedure, which is based on a process of oversimplification and correction. Experiments show that with our approach very intuitive models can be learned.