Real-time interactive sequence generation and control with Recurrent Neural Network ensembles
This addresses the need for live creative expression tools by allowing users to steer sequence generation in real-time, though it is incremental as it builds on existing RNN techniques.
The paper tackled the problem of enabling real-time interactive control in sequence generation with RNNs, proposing a method using RNN ensembles and dynamic mixture weights, and demonstrated it with character-based LSTM networks and a gestural interface for text generation.
Recurrent Neural Networks (RNN), particularly Long Short Term Memory (LSTM) RNNs, are a popular and very successful method for learning and generating sequences. However, current generative RNN techniques do not allow real-time interactive control of the sequence generation process, thus aren't well suited for live creative expression. We propose a method of real-time continuous control and 'steering' of sequence generation using an ensemble of RNNs and dynamically altering the mixture weights of the models. We demonstrate the method using character based LSTM networks and a gestural interface allowing users to 'conduct' the generation of text.