No LLM Solved Yu Tsumura's 554th Problem
This highlights a critical limitation in LLMs' problem-solving capabilities for mathematical reasoning, which is incremental as it focuses on a single unsolved case.
The authors identified a specific mathematical problem, Yu Tsumura's 554th problem, that no existing large language model (LLM) can solve, despite it being comparable in difficulty to IMO problems and having a publicly available solution likely in the models' training data.
We show, contrary to the optimism about LLM's problem-solving abilities, fueled by the recent gold medals that were attained, that a problem exists -- Yu Tsumura's 554th problem -- that a) is within the scope of an IMO problem in terms of proof sophistication, b) is not a combinatorics problem which has caused issues for LLMs, c) requires fewer proof techniques than typical hard IMO problems, d) has a publicly available solution (likely in the training data of LLMs), and e) that cannot be readily solved by any existing off-the-shelf LLM (commercial or open-source).