Rule Synergy Analysis using LLMs: State of the Art and Implications
This work addresses the problem of LLM reasoning about rule interactions for researchers, but it is incremental as it primarily evaluates existing models on a new dataset.
The paper investigated how well large language models (LLMs) understand complex rule interactions in dynamic environments like card games, using a dataset from Slay the Spire, and found that while LLMs excel at identifying non-synergistic pairs, they struggle with detecting positive and negative synergies.
Large language models (LLMs) have demonstrated strong performance across a variety of domains, including logical reasoning, mathematics, and more. In this paper, we investigate how well LLMs understand and reason about complex rule interactions in dynamic environments, such as card games. We introduce a dataset of card synergies from the game Slay the Spire, where pairs of cards are classified based on their positive, negative, or neutral interactions. Our evaluation shows that while LLMs excel at identifying non-synergistic pairs, they struggle with detecting positive and, particularly, negative synergies. We categorize common error types, including issues with timing, defining game states, and following game rules. Our findings suggest directions for future research to improve model performance in predicting the effect of rules and their interactions.