LIFI: Towards Linguistically Informed Frame Interpolation
This work addresses the challenge of generating accurate speech video interpolations for online communication, but it is incremental as it focuses on evaluation rather than a new method.
The paper tackles the problem of frame interpolation for speech videos by proposing linguistically-informed metrics and datasets to evaluate computer vision models, revealing that existing models fail to produce faithful interpolation despite high performance on conventional metrics.
In this work, we explore a new problem of frame interpolation for speech videos. Such content today forms the major form of online communication. We try to solve this problem by using several deep learning video generation algorithms to generate the missing frames. We also provide examples where computer vision models despite showing high performance on conventional non-linguistic metrics fail to accurately produce faithful interpolation of speech. With this motivation, we provide a new set of linguistically-informed metrics specifically targeted to the problem of speech videos interpolation. We also release several datasets to test computer vision video generation models of their speech understanding.