Context, Attention and Audio Feature Explorations for Audio Visual Scene-Aware Dialog
This work addresses improving multimodal dialog systems for Intelligent Virtual Assistants, but it is incremental as it builds on existing DSTC7 benchmarks with specific feature explorations.
The paper tackles the Audio Visual Scene-Aware Dialog (AVSD) task by exploring dialog topics as contextual features and multimodal attention, incorporating an end-to-end audio classification ConvNet (AclNet). The result shows that some model variations outperform the baseline system presented for the task.
With the recent advancements in AI, Intelligent Virtual Assistants (IVA) have become a ubiquitous part of every home. Going forward, we are witnessing a confluence of vision, speech and dialog system technologies that are enabling the IVAs to learn audio-visual groundings of utterances and have conversations with users about the objects, activities and events surrounding them. As a part of the 7th Dialog System Technology Challenges (DSTC7), for Audio Visual Scene-Aware Dialog (AVSD) track, We explore `topics' of the dialog as an important contextual feature into the architecture along with explorations around multimodal Attention. We also incorporate an end-to-end audio classification ConvNet, AclNet, into our models. We present detailed analysis of the experiments and show that some of our model variations outperform the baseline system presented for this task.