Neuron-Level Emotion Control in Speech-Generative Large Audio-Language Models
This work addresses the challenge of precise emotion control in speech-generative AI models, which is crucial for applications like virtual assistants and entertainment, though it is incremental as it builds on existing neuron-level analysis methods.
The paper tackled the problem of unreliable emotion control in large audio-language models for speech generation, which often leads to missed target affects and degraded linguistic fidelity. It demonstrated that compact emotion-sensitive neurons identified via success-filtered activation aggregation enable training-free emotion steering, yielding emotion-specific gains across three models as supported by automatic and human evaluation.
Large audio-language models (LALMs) can produce expressive speech, yet reliable emotion control remains elusive: conversions often miss the target affect and may degrade linguistic fidelity through refusals, hallucinations, or paraphrase. We present, to our knowledge, the first neuron-level study of emotion control in speech-generative LALMs and demonstrate that compact emotion-sensitive neurons (ESNs) are causally actionable, enabling training-free emotion steering at inference time. ESNs are identified via success-filtered activation aggregation enforcing both emotion realization and content preservation. Across three LALMs (Qwen2.5-Omni-7B, MiniCPM-o 4.5, Kimi-Audio), ESN interventions yield emotion-specific gains that generalize to unseen speakers and are supported by automatic and human evaluation. Controllability depends on selector design, mask sparsity, filtering, and intervention strength. Our results establish a mechanistic framework for training-free emotion control in speech generation.