On the rate of convergence of a classifier based on a Transformer encoder
This provides theoretical guarantees for Transformer classifiers in high-dimensional pattern recognition, though it is incremental as it builds on existing methods.
The paper analyzes the convergence rate of a Transformer encoder-based classifier's misclassification probability towards the optimal, showing it can avoid the curse of dimensionality under a hierarchical composition model for the posterior probability.
Pattern recognition based on a high-dimensional predictor is considered. A classifier is defined which is based on a Transformer encoder. The rate of convergence of the misclassification probability of the classifier towards the optimal misclassification probability is analyzed. It is shown that this classifier is able to circumvent the curse of dimensionality provided the aposteriori probability satisfies a suitable hierarchical composition model. Furthermore, the difference between Transformer classifiers analyzed theoretically in this paper and Transformer classifiers used nowadays in practice are illustrated by considering classification problems in natural language processing.