Step-Audio 2 Technical Report
This addresses the need for industry-strength audio and speech AI systems, offering enhanced responsiveness to paralinguistic information and reduced hallucination, though it appears incremental as it builds on existing multi-modal and retrieval-augmented approaches.
The paper tackles the problem of audio understanding and speech conversation by developing Step-Audio 2, an end-to-end multi-modal large language model that integrates latent audio encoding, reasoning-centric reinforcement learning, and retrieval-augmented generation, achieving state-of-the-art performance on various benchmarks compared to other solutions.
This paper presents Step-Audio 2, an end-to-end multi-modal large language model designed for industry-strength audio understanding and speech conversation. By integrating a latent audio encoder and reasoning-centric reinforcement learning (RL), Step-Audio 2 achieves promising performance in automatic speech recognition (ASR) and audio understanding. To facilitate genuine end-to-end speech conversation, Step-Audio 2 incorporates the generation of discrete audio tokens into language modeling, significantly enhancing its responsiveness to paralinguistic information such as speaking styles and emotions. To effectively leverage the rich textual and acoustic knowledge in real-world data, Step-Audio 2 integrates retrieval-augmented generation (RAG) and is able to call external tools such as web search to mitigate hallucination and audio search to switch timbres. Trained on millions of hours of speech and audio data, Step-Audio 2 delivers intelligence and expressiveness across diverse conversational scenarios. Evaluation results demonstrate that Step-Audio 2 achieves state-of-the-art performance on various audio understanding and conversational benchmarks compared to other open-source and commercial solutions. Please visit https://github.com/stepfun-ai/Step-Audio2 for more information.