A Multi-dimensional Evaluation of Tokenizer-free Multilingual Pretrained Models
This work addresses the practical deployment challenges for researchers and engineers in multilingual NLP, though it is incremental as it builds on existing tokenizer-free methods.
The paper tackled the problem of evaluating tokenizer-free multilingual pretrained models beyond accuracy, considering practical factors like memory usage and inference speed, and found that subword-based models often remain more practical with better performance and lower latency.
Recent work on tokenizer-free multilingual pretrained models show promising results in improving cross-lingual transfer and reducing engineering overhead (Clark et al., 2022; Xue et al., 2022). However, these works mainly focus on reporting accuracy on a limited set of tasks and data settings, placing less emphasis on other important factors when tuning and deploying the models in practice, such as memory usage, inference speed, and fine-tuning data robustness. We attempt to fill this gap by performing a comprehensive empirical comparison of multilingual tokenizer-free and subword-based models considering these various dimensions. Surprisingly, we find that subword-based models might still be the most practical choice in many settings, achieving better performance for lower inference latency and memory usage. Based on these results, we encourage future work in tokenizer-free methods to consider these factors when designing and evaluating new models.