Efficient Representations for Life-Long Learning and Autoencoding
This work addresses the challenge of building life-long learning systems that can reuse insights across tasks, which is a foundational goal in AI, but it appears incremental as it builds on existing theoretical formulations.
The paper tackles the problem of life-long learning by developing efficient algorithms for learning new internal representations that capture commonalities among target functions, enabling more efficient learning of subsequent tasks with less data. It provides algorithms for two types of commonalities: low-dimensional subspaces and nonlinear Boolean feature combinations, with the latter also offering an efficient method for constructing near-optimal sparse Boolean autoencoders.
It has been a long-standing goal in machine learning, as well as in AI more generally, to develop life-long learning systems that learn many different tasks over time, and reuse insights from tasks learned, "learning to learn" as they do so. In this work we pose and provide efficient algorithms for several natural theoretical formulations of this goal. Specifically, we consider the problem of learning many different target functions over time, that share certain commonalities that are initially unknown to the learning algorithm. Our aim is to learn new internal representations as the algorithm learns new target functions, that capture this commonality and allow subsequent learning tasks to be solved more efficiently and from less data. We develop efficient algorithms for two very different kinds of commonalities that target functions might share: one based on learning common low-dimensional and unions of low-dimensional subspaces and one based on learning nonlinear Boolean combinations of features. Our algorithms for learning Boolean feature combinations additionally have a dual interpretation, and can be viewed as giving an efficient procedure for constructing near-optimal sparse Boolean autoencoders under a natural "anchor-set" assumption.