Dance Dance Convolution
This addresses the need for custom choreography in rhythm games, offering an automated solution for players and creators, though it is incremental as it builds on existing neural network techniques.
The paper tackles the problem of automatically generating step charts for Dance Dance Revolution from raw audio, decomposing it into step placement and selection tasks. They propose a hybrid recurrent-convolutional network for placement and a conditional LSTM for selection, with the LSTM outperforming baseline methods.
Dance Dance Revolution (DDR) is a popular rhythm-based video game. Players perform steps on a dance platform in synchronization with music as directed by on-screen step charts. While many step charts are available in standardized packs, players may grow tired of existing charts, or wish to dance to a song for which no chart exists. We introduce the task of learning to choreograph. Given a raw audio track, the goal is to produce a new step chart. This task decomposes naturally into two subtasks: deciding when to place steps and deciding which steps to select. For the step placement task, we combine recurrent and convolutional neural networks to ingest spectrograms of low-level audio features to predict steps, conditioned on chart difficulty. For step selection, we present a conditional LSTM generative model that substantially outperforms n-gram and fixed-window approaches.