Distill2Vec: Dynamic Graph Representation Learning with Knowledge Distillation
This work addresses efficiency issues for practitioners using dynamic graph models, though it is incremental as it applies existing knowledge distillation to a specific domain.
The paper tackles the problem of high online inference latency and large parameter requirements in dynamic graph representation learning by proposing Distill2Vec, a knowledge distillation strategy that trains a compact model, achieving up to 5% relative gains in link prediction accuracy and a compression ratio of up to 7:100 compared to baselines.
Dynamic graph representation learning strategies are based on different neural architectures to capture the graph evolution over time. However, the underlying neural architectures require a large amount of parameters to train and suffer from high online inference latency, that is several model parameters have to be updated when new data arrive online. In this study we propose Distill2Vec, a knowledge distillation strategy to train a compact model with a low number of trainable parameters, so as to reduce the latency of online inference and maintain the model accuracy high. We design a distillation loss function based on Kullback-Leibler divergence to transfer the acquired knowledge from a teacher model trained on offline data, to a small-size student model for online data. Our experiments with publicly available datasets show the superiority of our proposed model over several state-of-the-art approaches with relative gains up to 5% in the link prediction task. In addition, we demonstrate the effectiveness of our knowledge distillation strategy, in terms of number of required parameters, where Distill2Vec achieves a compression ratio up to 7:100 when compared with baseline approaches. For reproduction purposes, our implementation is publicly available at https://stefanosantaris.github.io/Distill2Vec.