Learning to Make Predictions In Partially Observable Environments Without a Generative Model
This addresses the problem of model complexity in partially observable environments for researchers and practitioners in reinforcement learning and AI, offering a more efficient alternative to generative models.
The paper tackles the challenge of learning models in partially observable environments without requiring full generative models, by introducing prediction profile models that focus on a limited set of predictions. It demonstrates empirically that these models can learn important predictions in complex systems where standard generative models fail.
When faced with the problem of learning a model of a high-dimensional environment, a common approach is to limit the model to make only a restricted set of predictions, thereby simplifying the learning problem. These partial models may be directly useful for making decisions or may be combined together to form a more complete, structured model. However, in partially observable (non-Markov) environments, standard model-learning methods learn generative models, i.e. models that provide a probability distribution over all possible futures (such as POMDPs). It is not straightforward to restrict such models to make only certain predictions, and doing so does not always simplify the learning problem. In this paper we present prediction profile models: non-generative partial models for partially observable systems that make only a given set of predictions, and are therefore far simpler than generative models in some cases. We formalize the problem of learning a prediction profile model as a transformation of the original model-learning problem, and show empirically that one can learn prediction profile models that make a small set of important predictions even in systems that are too complex for standard generative models.