Learning a Deep Generative Model like a Program: the Free Category Prior
This work addresses a foundational problem in AI by proposing a novel way for neural networks to achieve compositional learning, challenging the assumption that symbolic representations are necessary for such capabilities.
This paper tackles the problem of enabling neural networks to learn compositional concepts like programs, similar to how humans combine words to form new concepts. It introduces the free category prior, a formalism that allows neural networks to act as primitives within probabilistic programs, and demonstrates end-to-end learning of both program structure and model parameters.
Humans surpass the cognitive abilities of most other animals in our ability to "chunk" concepts into words, and then combine the words to combine the concepts. In this process, we make "infinite use of finite means", enabling us to learn new concepts quickly and nest concepts within each-other. While program induction and synthesis remain at the heart of foundational theories of artificial intelligence, only recently has the community moved forward in attempting to use program learning as a benchmark task itself. The cognitive science community has thus often assumed that if the brain has simulation and reasoning capabilities equivalent to a universal computer, then it must employ a serialized, symbolic representation. Here we confront that assumption, and provide a counterexample in which compositionality is expressed via network structure: the free category prior over programs. We show how our formalism allows neural networks to serve as primitives in probabilistic programs. We learn both program structure and model parameters end-to-end.