Cat, Rat, Meow: On the Alignment of Language Model and Human Term-Similarity Judgments
This provides a novel evaluation setting for probing semantic associations in language models, which is incremental but useful for researchers analyzing model behavior and representations.
The paper tackled the problem of evaluating how well language models align with human similarity judgments on a word triplet task, finding that even small models can achieve human-level alignment and that instruction tuning significantly improves agreement.
Small and mid-sized generative language models have gained increasing attention. Their size and availability make them amenable to being analyzed at a behavioral as well as a representational level, allowing investigations of how these levels interact. We evaluate 32 publicly available language models for their representational and behavioral alignment with human similarity judgments on a word triplet task. This provides a novel evaluation setting to probe semantic associations in language beyond common pairwise comparisons. We find that (1) even the representations of small language models can achieve human-level alignment, (2) instruction-tuned model variants can exhibit substantially increased agreement, (3) the pattern of alignment across layers is highly model dependent, and (4) alignment based on models' behavioral responses is highly dependent on model size, matching their representational alignment only for the largest evaluated models.