What makes math problems hard for reinforcement learning: a case study
This work addresses the problem of rare instance search in reinforcement learning for researchers and practitioners, with incremental contributions based on a specific mathematical case study.
The paper tackled the challenge of finding rare high-reward instances in reinforcement learning by using the Andrews-Curtis conjecture as a case study, resulting in algorithmic enhancements and a topological hardness measure with implications for search problems, while also resolving open mathematical questions such as length reducibility in the Akbulut-Kirby series and potential counterexamples in the Miller-Schupp series.
Using a long-standing conjecture from combinatorial group theory, we explore, from multiple perspectives, the challenges of finding rare instances carrying disproportionately high rewards. Based on lessons learned in the context defined by the Andrews-Curtis conjecture, we propose algorithmic enhancements and a topological hardness measure with implications for a broad class of search problems. As part of our study, we also address several open mathematical questions. Notably, we demonstrate the length reducibility of all but two presentations in the Akbulut-Kirby series (1981), and resolve various potential counterexamples in the Miller-Schupp series (1991), including three infinite subfamilies.