The language of sounds unheard: Exploring musical timbre semantics of large language models
This work explores how LLMs capture human sensory semantics, offering insights into AI perception but is incremental as it builds on existing psychophysical research.
The study investigated whether large language models (LLMs) like ChatGPT can organize perceptual semantics of musical timbre similarly to humans by having it rate instrument sounds on semantic scales, finding partial correlation with human ratings but robust agreement on dimensions like brightness and pitch height, with internal variability comparable to humans.
Semantic dimensions of sound have been playing a central role in understanding the nature of auditory sensory experience as well as the broader relation between perception, language, and meaning. Accordingly, and given the recent proliferation of large language models (LLMs), here we asked whether such models exhibit an organisation of perceptual semantics similar to those observed in humans. Specifically, we prompted ChatGPT, a chatbot based on a state-of-the-art LLM, to rate musical instrument sounds on a set of 20 semantic scales. We elicited multiple responses in separate chats, analogous to having multiple human raters. ChatGPT generated semantic profiles that only partially correlated with human ratings, yet showed robust agreement along well-known psychophysical dimensions of musical sounds such as brightness (bright-dark) and pitch height (deep-high). Exploratory factor analysis suggested the same dimensionality but different spatial configuration of a latent factor space between the chatbot and human ratings. Unexpectedly, the chatbot showed degrees of internal variability that were comparable in magnitude to that of human ratings. Our work highlights the potential of LLMs to capture salient dimensions of human sensory experience.