Aikyam: A Video Conferencing Utility for Deaf and Dumb
This addresses a specific problem for deaf and dumb users in remote communication, but it is incremental as it builds on existing video conferencing platforms and standard models.
The paper tackles communication barriers for deaf and dumb individuals in video conferencing by proposing a utility that generates semantically correct sentences from sign gestures using an LSTM model for classification and a T5-based model for sentence construction, with results including audio output and annotated user feeds to aid smooth communication.
With the advent of the pandemic, the use of video conferencing platforms as a means of communication has greatly increased and with it, so have the remote opportunities. The deaf and dumb have traditionally faced several issues in communication, but now the effect is felt more severely. This paper proposes an all-encompassing video conferencing utility that can be used with existing video conferencing platforms to address these issues. Appropriate semantically correct sentences are generated from the signer's gestures which would be interpreted by the system. Along with an audio to emit this sentence, the user's feed is also used to annotate the sentence. This can be viewed by all participants, thus aiding smooth communication with all parties involved. This utility utilizes a simple LSTM model for classification of gestures. The sentences are constructed by a t5 based model. In order to achieve the required data flow, a virtual camera is used.