L0 Regularization Based Neural Network Design and Compression
This work addresses complexity reduction for scaling AI to real-world, off-the-cloud applications, but it is incremental as it applies an existing regularization method to new contexts.
The paper tackled the problem of massive over-parameterization in deep neural networks, which leads to issues like adversarial susceptibility and high costs, by applying L0 regularization to reduce complexity and analyze trade-offs, showing it captures input saliency on MNIST and signal modulation datasets.
We consider complexity of Deep Neural Networks (DNNs) and their associated massive over-parameterization. Such over-parametrization may entail susceptibility to adversarial attacks, loss of interpretability and adverse Size, Weight and Power - Cost (SWaP-C) considerations. We ask if there are methodical ways (regularization) to reduce complexity and how can we interpret trade-off between desired metric and complexity of DNN. Reducing complexity is directly applicable to scaling of AI applications to real world problems (especially for off-the-cloud applications). We show that presence and evaluation of the knee of the tradeoff curve. We apply a form of L0 regularization to MNIST data and signal modulation classifications. We show that such regularization captures saliency in the input space as well.