LSTM-SAKT: LSTM-Encoded SAKT-like Transformer for Knowledge Tracing
This work addresses incremental improvements in knowledge tracing for educational technology, focusing on competition performance.
The paper tackled the limitations of Transformer-based models in knowledge tracing by proposing an LSTM-encoded SAKT-like Transformer to enhance query/key/value information and address inter-container leakage, achieving 2nd place in the Riiid! Answer Correctness Prediction Kaggle competition with 3395 teams.
This paper introduces the 2nd place solution for the Riiid! Answer Correctness Prediction in Kaggle, the world's largest data science competition website. This competition was held from October 16, 2020, to January 7, 2021, with 3395 teams and 4387 competitors. The main insights and contributions of this paper are as follows. (i) We pointed out existing Transformer-based models are suffering from a problem that the information which their query/key/value can contain is limited. To solve this problem, we proposed a method that uses LSTM to obtain query/key/value and verified its effectiveness. (ii) We pointed out 'inter-container' leakage problem, which happens in datasets where questions are sometimes served together. To solve this problem, we showed special indexing/masking techniques that are useful when using RNN-variants and Transformer. (iii) We found additional hand-crafted features are effective to overcome the limits of Transformer, which can never consider the samples older than the sequence length.