Concentration Inequalities for the Stochastic Optimization of Unbounded Objectives with Application to Denoising Score Matching
This work addresses a problem relevant to researchers and practitioners dealing with stochastic optimization, particularly those working with unbounded objective functions, such as in denoising score matching, offering an incremental improvement.
The authors tackled the problem of stochastic optimization of unbounded objective functions, deriving novel concentration inequalities that bound statistical error, and achieved results with specific applications to denoising score matching. Their work establishes the benefit of sample reuse in certain algorithms.
We derive novel concentration inequalities that bound the statistical error for a large class of stochastic optimization problems, focusing on the case of unbounded objective functions. Our derivations utilize the following tools: 1) A new form of McDiarmid's inequality that is based on sample dependent one component difference bounds and which leads to a novel uniform law of large numbers result for unbounded functions. 2) A Rademacher complexity bound for families of functions that satisfy an appropriate local Lipschitz property. As an application of these results, we derive statistical error bounds for denoising score matching (DSM), an application that inherently requires one to consider unbounded objective functions, even when the data distribution has bounded support. In addition, our results establish the benefit of sample reuse in algorithms that employ easily sampled auxiliary random variables in addition to the training data, e.g., as in DSM, which uses auxiliary Gaussian random variables.