Action Shapley: A Training Data Selection Metric for World Model in Reinforcement Learning
This work addresses the challenge of costly or impractical environment interactions in offline and model-based RL, offering an incremental improvement in training data selection.
The paper tackles the problem of selecting high-quality training data for world models in reinforcement learning by introducing Action Shapley, a metric that improves efficacy and interpretability. Empirical results show the proposed algorithm achieves over 80% computational efficiency gain and consistently outperforms ad-hoc selection in data-constrained real-world cases.
Numerous offline and model-based reinforcement learning systems incorporate world models to emulate the inherent environments. A world model is particularly important in scenarios where direct interactions with the real environment is costly, dangerous, or impractical. The efficacy and interpretability of such world models are notably contingent upon the quality of the underlying training data. In this context, we introduce Action Shapley as an agnostic metric for the judicious and unbiased selection of training data. To facilitate the computation of Action Shapley, we present a randomized dynamic algorithm specifically designed to mitigate the exponential complexity inherent in traditional Shapley value computations. Through empirical validation across five data-constrained real-world case studies, the algorithm demonstrates a computational efficiency improvement exceeding 80\% in comparison to conventional exponential time computations. Furthermore, our Action Shapley-based training data selection policy consistently outperforms ad-hoc training data selection.