Belief-State Query Policies for User-Aligned POMDPs
This work addresses user alignment in planning for partially observable settings, representing an incremental advance with novel theoretical analysis.
The paper tackles the problem of aligning agent behavior with user constraints in partially observable environments by introducing parameterized belief-state query policies, proving theoretical properties of the cost function and developing algorithms that converge to optimal user-aligned behavior.
Planning in real-world settings often entails addressing partial observability while aligning with users' requirements. We present a novel framework for expressing users' constraints and preferences about agent behavior in a partially observable setting using parameterized belief-state query (BSQ) policies in the setting of goal-oriented partially observable Markov decision processes (gPOMDPs). We present the first formal analysis of such constraints and prove that while the expected cost function of a parameterized BSQ policy w.r.t its parameters is not convex, it is piecewise constant and yields an implicit discrete parameter search space that is finite for finite horizons. This theoretical result leads to novel algorithms that optimize gPOMDP agent behavior with guaranteed user alignment. Analysis proves that our algorithms converge to the optimal user-aligned behavior in the limit. Empirical results show that parameterized BSQ policies provide a computationally feasible approach for user-aligned planning in partially observable settings.