Rollout Algorithms and Approximate Dynamic Programming for Bayesian Optimization and Sequential Estimation
This work provides a general framework for sequential estimation, potentially benefiting researchers in optimization and control, but it appears incremental as it builds on existing rollout methods.
The paper introduces a unifying approximate dynamic programming framework for sequential estimation problems, applying rollout algorithms to Bayesian optimization and optimal measurement selection, with examples including puzzles like Wordle and Mastermind.
We provide a unifying approximate dynamic programming framework that applies to a broad variety of problems involving sequential estimation. We consider first the construction of surrogate cost functions for the purposes of optimization, and we focus on the special case of Bayesian optimization, using the rollout algorithm and some of its variations. We then discuss the more general case of sequential estimation of a random vector using optimal measurement selection, and its application to problems of stochastic and adaptive control. We distinguish between adaptive control of deterministic and stochastic systems: the former are better suited for the use of rollout, while the latter are well suited for the use of rollout with certainty equivalence approximations. As an example of the deterministic case, we discuss sequential decoding problems, and a rollout algorithm for the approximate solution of the Wordle and Mastermind puzzles, recently developed in the paper [BBB22].