Randomized Value Functions via Posterior State-Abstraction Sampling
This work addresses the challenge of sample efficiency in reinforcement learning for multi-task environments, representing an incremental improvement by integrating uncertainty modeling from bandit literature.
The paper tackles the problem of efficiently uncovering latent state abstractions in reinforcement learning while seeking optimal behavior, proposing an algorithm that maintains uncertainty over structure and achieves substantial performance gains in multi-task settings with shared low-dimensional representations.
State abstraction has been an essential tool for dramatically improving the sample efficiency of reinforcement-learning algorithms. Indeed, by exposing and accentuating various types of latent structure within the environment, different classes of state abstraction have enabled improved theoretical guarantees and empirical performance. When dealing with state abstractions that capture structure in the value function, however, a standard assumption is that the true abstraction has been supplied or unrealistically computed a priori, leaving open the question of how to efficiently uncover such latent structure while jointly seeking out optimal behavior. Taking inspiration from the bandit literature, we propose that an agent seeking out latent task structure must explicitly represent and maintain its uncertainty over that structure as part of its overall uncertainty about the environment. We introduce a practical algorithm for doing this using two posterior distributions over state abstractions and abstract-state values. In empirically validating our approach, we find that substantial performance gains lie in the multi-task setting where tasks share a common, low-dimensional representation.