Regret-based Reward Elicitation for Markov Decision Processes
This addresses the challenge for users in AI and robotics who need to define rewards in MDPs, offering a more efficient approach, though it is incremental as it builds on existing regret and elicitation methods.
The paper tackles the problem of specifying reward functions in Markov decision processes, which is often difficult for users, by framing it as preference elicitation and using minimax regret to compute robust policies with partial reward information; empirical results show that regret-based reward elicitation effectively produces near-optimal policies without requiring full reward specification.
The specification of aMarkov decision process (MDP) can be difficult. Reward function specification is especially problematic; in practice, it is often cognitively complex and time-consuming for users to precisely specify rewards. This work casts the problem of specifying rewards as one of preference elicitation and aims to minimize the degree of precision with which a reward function must be specified while still allowing optimal or near-optimal policies to be produced. We first discuss how robust policies can be computed for MDPs given only partial reward information using the minimax regret criterion. We then demonstrate how regret can be reduced by efficiently eliciting reward information using bound queries, using regret-reduction as a means for choosing suitable queries. Empirical results demonstrate that regret-based reward elicitation offers an effective way to produce near-optimal policies without resorting to the precise specification of the entire reward function.