American Sign Language fingerspelling recognition from video: Methods for unrestricted recognition and signer-independence
It addresses the understudied challenge of fingerspelling recognition for ASL users, with incremental improvements in handling signer variation.
The paper tackles the problem of recognizing fingerspelled letters in American Sign Language from video, achieving up to 8% letter error rates in signer-dependent settings and 17% in signer-independent settings using segmental conditional random fields with neural network features.
In this thesis, we study the problem of recognizing video sequences of fingerspelled letters in American Sign Language (ASL). Fingerspelling comprises a significant but relatively understudied part of ASL, and recognizing it is challenging for a number of reasons: It involves quick, small motions that are often highly coarticulated; it exhibits significant variation between signers; and there has been a dearth of continuous fingerspelling data collected. In this work, we propose several types of recognition approaches, and explore the signer variation problem. Our best-performing models are segmental (semi-Markov) conditional random fields using deep neural network-based features. In the signer-dependent setting, our recognizers achieve up to about 8% letter error rates. The signer-independent setting is much more challenging, but with neural network adaptation we achieve up to 17% letter error rates.