The Subjectivity of Monoculture
This reframes monoculture evaluation as a context-dependent inference problem, which is incremental for researchers and practitioners in AI ethics and model assessment.
The paper tackles the problem of evaluating monoculture in machine learning models, arguing that it is subjective and depends on the choice of null model and context, with experiments showing drastically different inferences when using a null model with item difficulty compared to previous methods.
Machine learning models -- including large language models (LLMs) -- are often said to exhibit monoculture, where outputs agree strikingly often. But what does it actually mean for models to agree too much? We argue that this question is inherently subjective, relying on two key decisions. First, the analyst must specify a baseline null model for what "independence" should look like. This choice is inherently subjective, and as we show, different null models result in dramatically different inferences about excess agreement. Second, we show that inferences depend on the population of models and items under consideration. Models that seem highly correlated in one context may appear independent when evaluated on a different set of questions, or against a different set of peers. Experiments on two large-scale benchmarks validate our theoretical findings. For example, we find drastically different inferences when using a null model with item difficulty compared to previous works that do not. Together, our results reframe monoculture evaluation not as an absolute property of model behavior, but as a context-dependent inference problem.