RLLaVA: An RL-central Framework for Language and Vision Assistants
This provides a flexible and efficient framework for researchers to implement RL algorithms in vision-language tasks, though it is incremental as it builds on existing RL and VLM methods.
The paper tackles the challenge of efficiently training large vision-language models by introducing RLLaVA, an RL-central framework that decouples RL logic from architecture and execution, enabling resource-efficient training of 1B-7B models on common GPUs, such as training a 4B model on a single 24GB GPU, and shows improved performance over base models in experiments.
We present an RL-central framework for Language and Vision Assistants (RLLaVA) with its formulation of Markov decision process (MDP). RLLaVA decouples RL algorithmic logic from model architecture and distributed execution, supporting researchers in implementing new RL algorithms with minimal code, and to plug in a broad family of RL methods and vision-language models (VLMs) while remaining agnostic to specific training and inference engines. RLLaVA makes resource-efficient training of 1B--7B models feasible on common GPUs; notably, 4B-scale models can be trained end-to-end with full-parameter updates on a single 24GB GPU. Experiments on multi-modal and agentic tasks demonstrate that RLLaVA has task extensibility, and the models trained with it consistently improve performance over base models, competitive with other specially engineered RL frameworks. The code is available at https://github.com/TinyLoopX/RLLaVA.