Reliable algorithm selection for machine learning-guided design
This addresses the challenge for researchers and practitioners in computational design fields, like protein and RNA engineering, by providing a principled selection approach, though it is incremental as it builds on existing prediction-powered inference techniques.
The paper tackles the problem of selecting machine learning-guided design algorithms to ensure successful outcomes, such as designing proteins with high binding affinity, by proposing a method that reliably forecasts label distributions and guarantees high-probability success when density ratios are known.
Algorithms for machine learning-guided design, or design algorithms, use machine learning-based predictions to propose novel objects with desired property values. Given a new design task -- for example, to design novel proteins with high binding affinity to a therapeutic target -- one must choose a design algorithm and specify any hyperparameters and predictive and/or generative models involved. How can these decisions be made such that the resulting designs are successful? This paper proposes a method for design algorithm selection, which aims to select design algorithms that will produce a distribution of design labels satisfying a user-specified success criterion -- for example, that at least ten percent of designs' labels exceed a threshold. It does so by combining designs' predicted property values with held-out labeled data to reliably forecast characteristics of the label distributions produced by different design algorithms, building upon techniques from prediction-powered inference. The method is guaranteed with high probability to return design algorithms that yield successful label distributions (or the null set if none exist), if the density ratios between the design and labeled data distributions are known. We demonstrate the method's effectiveness in simulated protein and RNA design tasks, in settings with either known or estimated density ratios.