Analysis and Design of Convolutional Networks via Hierarchical Tensor Decompositions
This work addresses the need for theoretical understanding in deep learning for researchers and practitioners, though it is incremental as it builds on existing methods.
The paper tackles the lack of formal analysis for the expressive power of convolutional networks by using hierarchical tensor decompositions to study their expressive efficiency and inductive bias, providing insights into architectural features like depth and width.
The driving force behind convolutional networks - the most successful deep learning architecture to date, is their expressive power. Despite its wide acceptance and vast empirical evidence, formal analyses supporting this belief are scarce. The primary notions for formally reasoning about expressiveness are efficiency and inductive bias. Expressive efficiency refers to the ability of a network architecture to realize functions that require an alternative architecture to be much larger. Inductive bias refers to the prioritization of some functions over others given prior knowledge regarding a task at hand. In this paper we overview a series of works written by the authors, that through an equivalence to hierarchical tensor decompositions, analyze the expressive efficiency and inductive bias of various convolutional network architectural features (depth, width, strides and more). The results presented shed light on the demonstrated effectiveness of convolutional networks, and in addition, provide new tools for network design.