On the Brittleness of LLMs: A Journey around Set Membership
This work highlights a critical reliability issue for users of LLMs, showing that even basic reasoning tasks can expose fragmented understanding, making it an incremental but important contribution to LLM evaluation.
The paper tackled the paradox of LLMs excelling at complex reasoning but failing on simple set membership queries, revealing through large-scale experiments that their performance is consistently brittle and unpredictable across various dimensions.
Large language models (LLMs) achieve superhuman performance on complex reasoning tasks, yet often fail on much simpler problems, raising concerns about their reliability and interpretability. We investigate this paradox through a focused study with two key design features: simplicity, to expose basic failure modes, and scale, to enable comprehensive controlled experiments. We focus on set membership queries -- among the most fundamental forms of reasoning -- using tasks like ``Is apple an element of the set \{pear, plum, apple, raspberry\}?''. We conduct a systematic empirical evaluation across prompt phrasing, semantic structure, element ordering, and model choice. Our large-scale analysis reveals that LLM performance on this elementary task is consistently brittle, and unpredictable across all dimensions, suggesting that the models' ``understanding'' of the set concept is fragmented and convoluted at best. Our work demonstrates that the large-scale experiments enabled by the simplicity of the problem allow us to map and analyze the failure modes comprehensively, making this approach a valuable methodology for LLM evaluation in general.