ScaMo: Exploring the Scaling Law in Autoregressive Motion Generation Model
This work addresses the problem of efficiently scaling motion generation models for researchers and practitioners, though it is incremental as it extends known scaling principles to a new domain.
The paper tackled the unexplored application of scaling laws to motion generation by introducing a scalable framework, confirming for the first time that test loss follows logarithmic and power laws with compute budgets, and predicting optimal model parameters for a budget of 1e18 with validated results.
The scaling law has been validated in various domains, such as natural language processing (NLP) and massive computer vision tasks; however, its application to motion generation remains largely unexplored. In this paper, we introduce a scalable motion generation framework that includes the motion tokenizer Motion FSQ-VAE and a text-prefix autoregressive transformer. Through comprehensive experiments, we observe the scaling behavior of this system. For the first time, we confirm the existence of scaling laws within the context of motion generation. Specifically, our results demonstrate that the normalized test loss of our prefix autoregressive models adheres to a logarithmic law in relation to compute budgets. Furthermore, we also confirm the power law between Non-Vocabulary Parameters, Vocabulary Parameters, and Data Tokens with respect to compute budgets respectively. Leveraging the scaling law, we predict the optimal transformer size, vocabulary size, and data requirements for a compute budget of $1e18$. The test loss of the system, when trained with the optimal model size, vocabulary size, and required data, aligns precisely with the predicted test loss, thereby validating the scaling law.