PatternMonitor: a whole pipeline with a much higher level of automation for guessing Android lock pattern based on videos

Pattern lock is a general technique used to realize identity authentication and access authorization on mobile terminal devices such as Android platform devices, but it is vulnerable to the attack proposed by recent researches that exploit information leaked by users while drawing patterns. However, the existing attacks on pattern lock are environmentally sensitive, and rely heavily on manual work, which constrains the practicability of these attack approaches. To attain a more practical attack, this paper designs the PatternMonitor, a whole pipeline with a much higher level of automation system againsts pattern lock, which extracts the guessed candidate patterns from a video containing pattern drawing: instead of manually cutting the target video and setting thresholds, it first employs recognition models to locate the target phone and keypoints of pattern drawing hand, which enables the gesture can be recognized even when the fingertips are shaded. Then, we extract the frames from the video where the drawing starts and ends. These pre-processed frames are inputs of target tracking model to generate trajectories, and further transformed into possible candidate patterns by performing our designed algorithm. To the best of our knowledge, our work is the first attack system to generate candidate patterns by only relying on hand movement instead of accurate fingertips capture. The experimental results demonstrates that our work is as accurate as previous work, which gives more than 90\% success rate within 20 attempts.