Exploring the Function Space of Deep-Learning Machines
This work provides theoretical insights into deep learning architectures, but it is incremental as it builds on existing physics-inspired methods without introducing new practical applications.
The paper investigates the function space of deep-learning machines by analyzing entropy growth relative to a reference function, revealing layer-wise convergence and phase transitions as error increases.
The function space of deep-learning machines is investigated by studying growth in the entropy of functions of a given error with respect to a reference function, realized by a deep-learning machine. Using physics-inspired methods we study both sparsely and densely-connected architectures to discover a layer-wise convergence of candidate functions, marked by a corresponding reduction in entropy when approaching the reference function, gain insight into the importance of having a large number of layers, and observe phase transitions as the error increases.