Imitation Learning from Pixel-Level Demonstrations by HashReward
This addresses the challenge of imitation learning in high-dimensional environments like games, offering a novel solution for improved generalization, though it appears incremental as it builds on existing imitation learning frameworks.
The paper tackles the problem of imitation learning in high-dimensional pixel-based environments, where existing methods fail due to poor discriminator training, and proposes HashReward, which uses supervised hashing to balance dimensionality reduction and discriminative training, achieving significant performance improvements over state-of-the-art methods.
One of the key issues for imitation learning lies in making policy learned from limited samples to generalize well in the whole state-action space. This problem is much more severe in high-dimensional state environments, such as game playing with raw pixel inputs. Under this situation, even state-of-the-art adversary-based imitation learning algorithms fail. Through empirical studies, we find that the main cause lies in the failure of training a powerful discriminator to generate meaningful rewards in high-dimensional environments. Although it seems that dimensionality reduction can help, a straightforward application of off-the-shelf methods cannot achieve good performance. In this work, we show in theory that the balance between dimensionality reduction and discriminative training is essential for effective learning. To achieve this target, we propose HashReward, which utilizes the idea of supervised hashing to realize such an ideal balance. Experimental results show that HashReward could outperform state-of-the-art methods for a large gap under the challenging high-dimensional environments.