Generalized Linear Markov Decision Process
This work addresses the problem of applying reinforcement learning in real-world domains like healthcare and e-commerce where rewards are nonlinear and data is scarce, representing an incremental extension of the linear MDP framework.
The paper tackles the limitation of linear MDPs by proposing the Generalized Linear MDP (GLMDP) framework, which models rewards using generalized linear models to handle nonlinear or discrete structures, and it develops offline RL algorithms that achieve theoretical guarantees on policy suboptimality and improved sample efficiency in data-scarce settings.
The linear Markov Decision Process (MDP) framework offers a principled foundation for reinforcement learning (RL) with strong theoretical guarantees and sample efficiency. However, its restrictive assumption-that both transition dynamics and reward functions are linear in the same feature space-limits its applicability in real-world domains, where rewards often exhibit nonlinear or discrete structures. Motivated by applications such as healthcare and e-commerce, where data is scarce and reward signals can be binary or count-valued, we propose the Generalized Linear MDP (GLMDP) framework-an extension of the linear MDP framework-that models rewards using generalized linear models (GLMs) while maintaining linear transition dynamics. We establish the Bellman completeness of GLMDPs with respect to a new function class that accommodates nonlinear rewards and develop two offline RL algorithms: Generalized Pessimistic Value Iteration (GPEVI) and a semi-supervised variant (SS-GPEVI) that utilizes both labeled and unlabeled trajectories. Our algorithms achieve theoretical guarantees on policy suboptimality and demonstrate improved sample efficiency in settings where reward labels are expensive or limited.