GenerationMania: Learning to Semantically Choreograph
This addresses a domain-specific problem for rhythm game developers and enthusiasts by automating chart creation, though it is incremental as it builds on existing neural network methods.
The paper tackles the problem of automatically generating game stages (charts) for the rhythm game Beatmania from arbitrary music, using deep neural networks to predict player-controlled sounds and map controls, achieving a significantly higher F1-score than LSTM baselines in reconstruction tasks.
Beatmania is a rhythm action game where players must reproduce some of the sounds of a song by pressing specific controller buttons at the correct time. In this paper we investigate the use of deep neural networks to automatically create game stages - called charts - for arbitrary pieces of music. Our technique uses a multi-layer feed-forward network trained on sound sequence summary statistics to predict which sounds in the music are to be played by the player and which will play automatically. We use another neural network along with rules to determine which controls should be mapped to which sounds. We evaluated our system on the ability to reconstruct charts in a held-out test set, achieving an $F_1$-score that significantly beats LSTM baselines.