Statistically Efficient Bayesian Sequential Experiment Design via Reinforcement Learning with Cross-Entropy Estimators
This work improves the efficiency of Bayesian sequential experiment design for researchers and practitioners in fields like machine learning and statistics, offering a more scalable and accurate method, though it is incremental as it builds on prior reinforcement learning approaches.
The paper tackled the problem of learning amortized design policies for sequential experiments by addressing the exponential sample complexity of existing expected information gain estimators, proposing a cross-entropy-based estimator that overcomes this limitation and enables learning superior policies across various design spaces and model types.
Reinforcement learning can learn amortised design policies for designing sequences of experiments. However, current amortised methods rely on estimators of expected information gain (EIG) that require an exponential number of samples on the magnitude of the EIG to achieve an unbiased estimation. We propose the use of an alternative estimator based on the cross-entropy of the joint model distribution and a flexible proposal distribution. This proposal distribution approximates the true posterior of the model parameters given the experimental history and the design policy. Our method overcomes the exponential-sample complexity of previous approaches and provide more accurate estimates of high EIG values. More importantly, it allows learning of superior design policies, and is compatible with continuous and discrete design spaces, non-differentiable likelihoods and even implicit probabilistic models.