A case for new neural network smoothness constraints
This paper highlights a foundational problem for the entire field of machine learning regarding the limitations of current smoothness constraints.
This paper argues that model smoothness is a useful inductive bias for improving generalization, adversarial robustness, generative modeling, and reinforcement learning. It identifies that current methods for imposing smoothness constraints are inflexible, fail to account for data modalities, and have poorly understood interactions with other model components.
How sensitive should machine learning models be to input changes? We tackle the question of model smoothness and show that it is a useful inductive bias which aids generalization, adversarial robustness, generative modeling and reinforcement learning. We explore current methods of imposing smoothness constraints and observe they lack the flexibility to adapt to new tasks, they don't account for data modalities, they interact with losses, architectures and optimization in ways not yet fully understood. We conclude that new advances in the field are hinging on finding ways to incorporate data, tasks and learning into our definitions of smoothness.