Rhapsody: A Dataset for Highlight Detection in Podcasts
This addresses the challenge of information access in long-form spoken media for podcast users, but it is incremental as it builds on existing methods with new data.
The authors tackled the problem of automatically detecting highlights in podcasts by introducing the Rhapsody dataset with 13K episodes and segment-level scores, finding that fine-tuned models using speech and transcript features significantly outperform zero-shot language models like GPT-4o and Gemini.
Podcasts have become daily companions for half a billion users. Given the enormous amount of podcast content available, highlights provide a valuable signal that helps viewers get the gist of an episode and decide if they want to invest in listening to it in its entirety. However, identifying highlights automatically is challenging due to the unstructured and long-form nature of the content. We introduce Rhapsody, a dataset of 13K podcast episodes paired with segment-level highlight scores derived from YouTube's 'most replayed' feature. We frame the podcast highlight detection as a segment-level binary classification task. We explore various baseline approaches, including zero-shot prompting of language models and lightweight fine-tuned language models using segment-level classification heads. Our experimental results indicate that even state-of-the-art language models like GPT-4o and Gemini struggle with this task, while models fine-tuned with in-domain data significantly outperform their zero-shot performance. The fine-tuned model benefits from leveraging both speech signal features and transcripts. These findings highlight the challenges for fine-grained information access in long-form spoken media.