Gujarati-English Code-Switching Speech Recognition using ensemble prediction of spoken language
This work addresses the challenge of recognizing language in code-switched speech for applications like multilingual communication, though it is incremental as it builds on existing methods without major performance gains.
The paper tackled the problem of improving code-switched speech recognition by conditioning transformer layers on language ID to better predict language from spoken data, but it did not significantly reduce word error rate (WER).
An important and difficult task in code-switched speech recognition is to recognize the language, as lots of words in two languages can sound similar, especially in some accents. We focus on improving performance of end-to-end Automatic Speech Recognition models by conditioning transformer layers on language ID of words and character in the output in an per layer supervised manner. To this end, we propose two methods of introducing language specific parameters and explainability in the multi-head attention mechanism, and implement a Temporal Loss that helps maintain continuity in input alignment. Despite being unable to reduce WER significantly, our method shows promise in predicting the correct language from just spoken data. We introduce regularization in the language prediction by dropping LID in the sequence, which helps align long repeated output sequences.