STCN: Stochastic Temporal Convolutional Networks
This work addresses a problem in machine learning for sequence modeling by offering a more expressive and robust alternative to existing methods, though it appears incremental as it builds on known concepts.
The authors tackled the performance gap between convolutional architectures and stochastic RNNs in sequence modeling by proposing STCNs, which combine temporal convolutional networks with stochastic latent spaces, achieving state-of-the-art log-likelihoods and high-quality long-range predictions in tasks like handwritten text modeling.
Convolutional architectures have recently been shown to be competitive on many sequence modelling tasks when compared to the de-facto standard of recurrent neural networks (RNNs), while providing computational and modeling advantages due to inherent parallelism. However, currently there remains a performance gap to more expressive stochastic RNN variants, especially those with several layers of dependent random variables. In this work, we propose stochastic temporal convolutional networks (STCNs), a novel architecture that combines the computational advantages of temporal convolutional networks (TCN) with the representational power and robustness of stochastic latent spaces. In particular, we propose a hierarchy of stochastic latent variables that captures temporal dependencies at different time-scales. The architecture is modular and flexible due to the decoupling of the deterministic and stochastic layers. We show that the proposed architecture achieves state of the art log-likelihoods across several tasks. Finally, the model is capable of predicting high-quality synthetic samples over a long-range temporal horizon in modeling of handwritten text.