Learning the Effects of Physical Actions in a Multi-modal Environment
This addresses a key limitation in AI planning for embodied agents, though it is incremental in extending existing multi-modal approaches.
The paper tackles the problem of LLMs' inadequate physical commonsense by introducing a multi-modal task to predict action outcomes from sensory inputs, showing that combining visual and textual information helps models generalize and learn physical commonsense reasoning better.
Large Language Models (LLMs) handle physical commonsense information inadequately. As a result of being trained in a disembodied setting, LLMs often fail to predict an action's outcome in a given environment. However, predicting the effects of an action before it is executed is crucial in planning, where coherent sequences of actions are often needed to achieve a goal. Therefore, we introduce the multi-modal task of predicting the outcomes of actions solely from realistic sensory inputs (images and text). Next, we extend an LLM to model latent representations of objects to better predict action outcomes in an environment. We show that multi-modal models can capture physical commonsense when augmented with visual information. Finally, we evaluate our model's performance on novel actions and objects and find that combining modalities help models to generalize and learn physical commonsense reasoning better.