A naive aggregation algorithm for improving generalization in a class of learning problems
This work addresses generalization in learning problems with expert advice, but it appears incremental as it builds on existing aggregation methods without clear broad impact.
The paper tackles the problem of improving generalization in learning with expert advice by proposing a naive aggregation algorithm that sequentially determines mixing strategies to achieve a consensus parameter estimate better than individual experts. Numerical results are presented for a nonlinear regression case, but no concrete performance numbers are provided.
In this brief paper, we present a naive aggregation algorithm for a typical learning problem with expert advice setting, in which the task of improving generalization, i.e., model validation, is embedded in the learning process as a sequential decision-making problem. In particular, we consider a class of learning problem of point estimations for modeling high-dimensional nonlinear functions, where a group of experts update their parameter estimates using the discrete-time version of gradient systems, with small additive noise term, guided by the corresponding subsample datasets obtained from the original dataset. Here, our main objective is to provide conditions under which such an algorithm will sequentially determine a set of mixing distribution strategies used for aggregating the experts' estimates that ultimately leading to an optimal parameter estimate, i.e., as a consensus solution for all experts, which is better than any individual expert's estimate in terms of improved generalization or learning performances. Finally, as part of this work, we present some numerical results for a typical case of nonlinear regression problem.