Efficient and Effective Implicit Dynamic Graph Neural Network
This work addresses a gap in dynamic graph learning for researchers and practitioners, offering a novel method with significant computational improvements, though it is incremental in extending implicit models to dynamic settings.
The paper tackles the problem of applying implicit graph neural networks to dynamic graphs, where long-range dependencies and oversmoothing are more pronounced, by proposing the first implicit dynamic graph neural network (IDGNN) with a theoretically guaranteed fixed-point representation and an efficient bilevel optimization training algorithm that achieves up to 1600x speed-up while maintaining performance.
Implicit graph neural networks have gained popularity in recent years as they capture long-range dependencies while improving predictive performance in static graphs. Despite the tussle between performance degradation due to the oversmoothing of learned embeddings and long-range dependency being more pronounced in dynamic graphs, as features are aggregated both across neighborhood and time, no prior work has proposed an implicit graph neural model in a dynamic setting. In this paper, we present Implicit Dynamic Graph Neural Network (IDGNN) a novel implicit neural network for dynamic graphs which is the first of its kind. A key characteristic of IDGNN is that it demonstrably is well-posed, i.e., it is theoretically guaranteed to have a fixed-point representation. We then demonstrate that the standard iterative algorithm often used to train implicit models is computationally expensive in our dynamic setting as it involves computing gradients, which themselves have to be estimated in an iterative manner. To overcome this, we pose an equivalent bilevel optimization problem and propose an efficient single-loop training algorithm that avoids iterative computation by maintaining moving averages of key components of the gradients. We conduct extensive experiments on real-world datasets on both classification and regression tasks to demonstrate the superiority of our approach over the state-of-the-art baselines. We also demonstrate that our bi-level optimization framework maintains the performance of the expensive iterative algorithm while obtaining up to \textbf{1600x} speed-up.