Nonparametric Online Learning Using Lipschitz Regularized Deep Neural Networks
This addresses the challenge of applying deep neural networks to online learning tasks with dependent data, which is incremental as it extends regularization techniques to a new setting.
The paper tackles the problem of online learning with non-i.i.d. data using Lipschitz regularized deep neural networks, proving that these networks guarantee convergence to the best possible prediction strategy.
Deep neural networks are considered to be state of the art models in many offline machine learning tasks. However, their performance and generalization abilities in online learning tasks are much less understood. Therefore, we focus on online learning and tackle the challenging problem where the underlying process is stationary and ergodic and thus removing the i.i.d. assumption and allowing observations to depend on each other arbitrarily. We prove the generalization abilities of Lipschitz regularized deep neural networks and show that by using those networks, a convergence to the best possible prediction strategy is guaranteed.