Monotonic Differentiable Sorting Networks
This addresses a specific issue in gradient-based optimization for tasks requiring sorting or ranking supervision, such as in machine learning models, but it is incremental as it builds on existing differentiable sorting methods.
The paper tackled the problem of non-monotonicity in differentiable sorting algorithms, which can cause incorrect gradient signs during optimization, by proposing a novel relaxation of conditional swap operations that guarantees monotonicity. The result is a family of sigmoid functions that produce differentiable sorting networks with improved performance over previous methods.
Differentiable sorting algorithms allow training with sorting and ranking supervision, where only the ordering or ranking of samples is known. Various methods have been proposed to address this challenge, ranging from optimal transport-based differentiable Sinkhorn sorting algorithms to making classic sorting networks differentiable. One problem of current differentiable sorting methods is that they are non-monotonic. To address this issue, we propose a novel relaxation of conditional swap operations that guarantees monotonicity in differentiable sorting networks. We introduce a family of sigmoid functions and prove that they produce differentiable sorting networks that are monotonic. Monotonicity ensures that the gradients always have the correct sign, which is an advantage in gradient-based optimization. We demonstrate that monotonic differentiable sorting networks improve upon previous differentiable sorting methods.