Quantitative Bounds for Sorting-Based Permutation-Invariant Embeddings
This work addresses theoretical gaps in permutation-invariant embeddings for graph deep learning, providing quantitative bounds that are incremental but important for understanding model robustness and efficiency.
The paper tackles the problem of determining optimal embedding dimensions and bi-Lipschitz constants for sorting-based permutation-invariant embeddings used in graph deep learning, achieving improved upper bounds for injectivity dimension and showing distortion bounds that depend quadratically on n and are independent of d.
We study the sorting-based embedding $β_{\mathbf A} : \mathbb R^{n \times d} \to \mathbb R^{n \times D}$, $\mathbf X \mapsto {\downarrow}(\mathbf X \mathbf A)$, where $\downarrow$ denotes column wise sorting of matrices. Such embeddings arise in graph deep learning where outputs should be invariant to permutations of graph nodes. Previous work showed that for large enough $D$ and appropriate $\mathbf A$, the mapping $β_{\mathbf A}$ is injective, and moreover satisfies a bi-Lipschitz condition. However, two gaps remain: firstly, the optimal size $D$ required for injectivity is not yet known, and secondly, no estimates of the bi-Lipschitz constants of the mapping are known. In this paper, we make substantial progress in addressing both of these gaps. Regarding the first gap, we improve upon the best known upper bounds for the embedding dimension $D$ necessary for injectivity, and also provide a lower bound on the minimal injectivity dimension. Regarding the second gap, we construct matrices $\mathbf A$, so that the bi-Lipschitz distortion of $β_{\mathbf A} $ depends quadratically on $n$, and is completely independent of $d$. We also show that the distortion of $β_{\mathbf A}$ is necessarily at least in $Ω(\sqrt{n})$. Finally, we provide similar results for variants of $β_{\mathbf A}$ obtained by applying linear projections to reduce the output dimension of $β_{\mathbf A}$.