How to Backpropagate through Hungarian in Your DETR?
This solves a gradient issue in DETR for object detection, enabling more stable training, but it is incremental as it builds on existing DETR frameworks.
The paper addresses the misalignment between assignment cost and global loss in DETR, proposing a method to express the loss as assignment-dependent and independent terms, enabling proper backpropagation through the Hungarian algorithm. Experiments show improved convergence and potential performance gains.
The DEtection TRansformer (DETR) approach, which uses a transformer encoder-decoder architecture and a set-based global loss, has become a building block in many transformer based applications. However, as originally presented, the assignment cost and the global loss are not aligned, i.e., reducing the former is likely but not guaranteed to reduce the latter. And the issue of gradient is ignored when a combinatorial solver such as Hungarian is used. In this paper we show that the global loss can be expressed as the sum of an assignment-independent term, and an assignment-dependent term which can be used to define the assignment cost matrix. Recent results on generalized gradients of optimal assignment cost with respect to parameters of an assignment problem are then used to define generalized gradients of the loss with respect to network parameters, and backpropagation is carried out properly. Our experiments using the same loss weights show interesting convergence properties and a potential for further performance improvements.