Iterative Foundation Model Fine-Tuning on Multiple Rewards
This addresses the need for multi-criteria optimization in applications like text generation and drug discovery, representing an incremental improvement over existing RL fine-tuning methods.
The paper tackles the problem of fine-tuning foundation models with multiple reward signals, proposing an iterative RL-based method that outperforms state-of-the-art baselines across text, biological sequence, and small molecule generation domains.
Fine-tuning foundation models has emerged as a powerful approach for generating objects with specific desired properties. Reinforcement learning (RL) provides an effective framework for this purpose, enabling models to generate outputs that maximize a given reward function. However, in many applications such as text generation and drug discovery, it can be suboptimal to optimize using a single reward signal, as multiple evaluation criteria are often necessary. This paper proposes a novel reinforcement learning-based method for fine-tuning foundation models using multiple reward signals. By employing an iterative fine-tuning strategy across these rewards, our approach generalizes state-of-the-art RL-based methods. We further provide a theoretical analysis that offers insights into the performance of multi-reward RL fine-tuning. Experimental results across diverse domains including text, biological sequence, and small molecule generation, demonstrate the effectiveness of the proposed algorithm compared to state-of-the-art baselines.