Falin Hei

Qi Ju, Falin Hei, Zhemei Fang et al.

Reinforcement Learning (RL) heavily relies on the careful design of the reward function. However, accurately assigning rewards to each state-action pair in Long-Term Reinforcement Learning (LTRL) tasks remains a significant challenge. As a result, RL agents are often trained under expert guidance. Inspired by the ordinal utility theory in economics, we propose a novel reward estimation algorithm: ELO-Rating based Reinforcement Learning (ERRL). This approach features two key contributions. First, it uses expert preferences over trajectories rather than cardinal rewards (utilities) to compute the ELO rating of each trajectory as its reward. Second, a new reward redistribution algorithm is introduced to alleviate training instability in the absence of a fixed anchor reward. In long-term scenarios (up to 5000 steps), where traditional RL algorithms struggle, our method outperforms several state-of-the-art baselines. Additionally, we conduct a comprehensive analysis of how expert preferences influence the results.

Ju Qi, Falin Hei, Ting Feng et al.

Counterfactual Regret Minimization (CFR) and its variants are widely recognized as effective algorithms for solving extensive-form imperfect information games. Recently, many improvements have been focused on enhancing the convergence speed of the CFR algorithm. However, most of these variants are not applicable under Monte Carlo (MC) conditions, making them unsuitable for training in large-scale games. We introduce a new MC-based algorithm for solving extensive-form imperfect information games, called MCCFVFP (Monte Carlo Counterfactual Value-Based Fictitious Play). MCCFVFP combines CFR's counterfactual value calculations with fictitious play's best response strategy, leveraging the strengths of fictitious play to gain significant advantages in games with a high proportion of dominated strategies. Experimental results show that MCCFVFP achieved convergence speeds approximately 20\%$\sim$50\% faster than the most advanced MCCFR variants in games like poker and other test games.