Deriving the Scaled-Dot-Function via Maximum Likelihood Estimation and Maximum Entropy Approach
This provides a theoretical explanation for a key component in transformer models, which is incremental as it offers new insights without introducing a new method.
The paper tackles the problem of explaining the scaled-dot-product function in transformers by deriving it using maximum likelihood estimation and a maximum entropy approach, modeling sequences as Gaussian distributions with variances and means dependent on time steps and vectors.
In this paper, we present a maximum likelihood estimation approach to determine the value vector in transformer models. We model the sequence of value vectors, key vectors, and the query vector as a sequence of Gaussian distributions. The variance in each Gaussian distribution depends on the time step, the corresponding key vector, and the query vector. The mean value in each Gaussian distribution depends on the time step, and the corresponding value vector. This analysis may offer a new explanation of the scaled-dot-product function or softmax function used in transformer architectures [1]. Another explanation, inspired by [4], is based on the maximum entropy approach in natural language processing [5]. In this approach, a query vector and key vectors are used to derive the feature functions for the maximum entropy model.