Log2Motion: Biomechanical Motion Synthesis from Touch Logs
This work addresses the challenge of understanding ergonomics and motor control in touch interactions for researchers and developers, representing a novel computational problem rather than an incremental improvement.
The paper tackles the problem of synthesizing biomechanically plausible motion from touch logs on mobile devices, which typically lack interaction details, by proposing a reinforcement learning-driven musculoskeletal simulation that integrates with a software emulator. The result is Log2Motion, which generates motion sequences consistent with touch logs and is validated against human motion capture data and a large-scale dataset, providing estimates of motion, speed, accuracy, and effort.
Touch data from mobile devices are collected at scale but reveal little about the interactions that produce them. While biomechanical simulations can illuminate motor control processes, they have not yet been developed for touch interactions. To close this gap, we propose a novel computational problem: synthesizing plausible motion directly from logs. Our key insight is a reinforcement learning-driven musculoskeletal forward simulation that generates biomechanically plausible motion sequences consistent with events recorded in touch logs. We achieve this by integrating a software emulator into a physics simulator, allowing biomechanical models to manipulate real applications in real-time. Log2Motion produces rich syntheses of user movements from touch logs, including estimates of motion, speed, accuracy, and effort. We assess the plausibility of generated movements by comparing against human data from a motion capture study and prior findings, and demonstrate Log2Motion in a large-scale dataset. Biomechanical motion synthesis provides a new way to understand log data, illuminating the ergonomics and motor control underlying touch interactions.