Differentiable Grammars for Videos
This addresses the need for interpretable generative models in video analysis, offering a novel approach that is not incremental but introduces a new method for a known bottleneck.
The paper tackles the problem of learning formal regular grammars from continuous video data to capture sequential structures, resulting in a model that outperforms state-of-the-art methods on challenging datasets and improves accuracy for forecasting future activities.
This paper proposes a novel algorithm which learns a formal regular grammar from real-world continuous data, such as videos. Learning latent terminals, non-terminals, and production rules directly from continuous data allows the construction of a generative model capturing sequential structures with multiple possibilities. Our model is fully differentiable, and provides easily interpretable results which are important in order to understand the learned structures. It outperforms the state-of-the-art on several challenging datasets and is more accurate for forecasting future activities in videos. We plan to open-source the code. https://sites.google.com/view/differentiable-grammars