A Generalist Agent
This work addresses the problem of developing versatile AI systems for broad applications, representing a step towards more integrated and flexible agents.
The authors tackled the challenge of creating a single generalist agent capable of handling diverse tasks across modalities and embodiments, resulting in Gato, a model that can perform activities like playing Atari, captioning images, chatting, and controlling a robot arm with the same network weights.
Inspired by progress in large-scale language modeling, we apply a similar approach towards building a single generalist agent beyond the realm of text outputs. The agent, which we refer to as Gato, works as a multi-modal, multi-task, multi-embodiment generalist policy. The same network with the same weights can play Atari, caption images, chat, stack blocks with a real robot arm and much more, deciding based on its context whether to output text, joint torques, button presses, or other tokens. In this report we describe the model and the data, and document the current capabilities of Gato.