Scaling Probabilistic Transformer via Efficient Cross-Scale Hyperparameter Transfer
This work addresses the scaling difficulty of probabilistic models for NLP practitioners, enabling larger deployment of PT with reduced tuning cost.
The authors scaled Probabilistic Transformer (PT) to 0.4B parameters by applying Maximal Update Parametrization (muP) to enable hyperparameter transfer from small models. PT outperforms standard Transformers on Masked Language Modeling tasks under the same parameter budget.
Probabilistic Transformer (PT), a white-box probabilistic model for contextual word representation, has demonstrated substantial similarity to standard Transformers in both computational structure and downstream task performance on small models and small to medium sized datasets. However, PT is less robust to hyperparameter choices than standard Transformers, making it harder to scale efficiently. In this work, we follow Maximal Update Parametrization (muP) to rescale PT's parameters, so that hyperparameters optimized on small models can be transferred to larger models without additional tuning. With this approach, we successfully scale PT to models with up to 0.4B parameters. Experiments show that PT consistently outperforms standard transformer under the same parameter budget on Masked Language Modeling (MLM) tasks. We hope this work will contribute to the practical deployment of probabilistic models at substantially larger scales in the future.