Light Differentiable Logic Gate Networks
This work solves scaling issues for DLGNs, which are efficient at inference, making them more practical for deployment, though it is incremental as it builds on existing parametrization methods.
The paper tackled the problem of scaling differentiable logic gate networks (DLGNs) by addressing vanishing gradients, discretization errors, and high training costs through a reparametrization of logic gate neurons, resulting in a 4x reduction in model size, up to 1.86x speedup in backward pass, and 8.5x fewer training steps while maintaining or improving accuracy on CIFAR-100.
Differentiable logic gate networks (DLGNs) exhibit extraordinary efficiency at inference while sustaining competitive accuracy. But vanishing gradients, discretization errors, and high training cost impede scaling these networks. Even with dedicated parameter initialization schemes from subsequent works, increasing depth still harms accuracy. We show that the root cause of these issues lies in the underlying parametrization of logic gate neurons themselves. To overcome this issue, we propose a reparametrization that also shrinks the parameter size logarithmically in the number of inputs per gate. For binary inputs, this already reduces the model size by 4x, speeds up the backward pass by up to 1.86x, and converges in 8.5x fewer training steps. On top of that, we show that the accuracy on CIFAR-100 remains stable and sometimes superior to the original parametrization.