Feature Qualification by Deep Nets: A Constructive Approach
This work addresses the challenge of feature qualification in deep learning, offering a theoretical framework for understanding which features are important in specific tasks, though it appears incremental as it builds on existing deep net properties.
The paper tackles the problem of identifying crucial data features in learning tasks by constructing a linear deep net operator with sigmoid activation that achieves optimal approximation performance for smooth and radial functions, and provides theoretical evidence for qualifying features like smoothness and radialness.
The great success of deep learning has stimulated avid research activities in verifying the power of depth in theory, a common consensus of which is that deep net are versatile in approximating and learning numerous functions. Such a versatility certainly enhances the understanding of the power of depth, but makes it difficult to judge which data features are crucial in a specific learning task. This paper proposes a constructive approach to equip deep nets for the feature qualification purpose. Using the product-gate nature and localized approximation property of deep nets with sigmoid activation (deep sigmoid nets), we succeed in constructing a linear deep net operator that possesses optimal approximation performance in approximating smooth and radial functions. Furthermore, we provide theoretical evidences that the constructed deep net operator is capable of qualifying multiple features such as the smoothness and radialness of the target functions.