Cliqueformer: Model-Based Optimization with Structured Transformers
This work addresses offline model-based optimization for design problems, offering a novel method that improves performance in specific domains like chemical and genetic design.
The paper tackled the problem of applying large neural networks to design tasks like protein engineering by introducing Cliqueformer, a transformer-based architecture that learns structure through functional graphical models to address distribution shift, achieving superior performance across domains such as chemical and genetic design.
Large neural networks excel at prediction tasks, but their application to design problems, such as protein engineering or materials discovery, requires solving offline model-based optimization (MBO) problems. While predictive models may not directly translate to effective design, recent MBO algorithms incorporate reinforcement learning and generative modeling approaches. Meanwhile, theoretical work suggests that exploiting the target function's structure can enhance MBO performance. We present Cliqueformer, a transformer-based architecture that learns the black-box function's structure through functional graphical models (FGM), addressing distribution shift without relying on explicit conservative approaches. Across various domains, including chemical and genetic design tasks, Cliqueformer demonstrates superior performance compared to existing methods.