Advancing Constrained Monotonic Neural Networks: Achieving Universal Approximation Beyond Bounded Activations
This work addresses optimization challenges in constrained monotonic neural networks, providing theoretical grounding and architectural simplification for applications requiring monotonicity, such as in fairness or interpretability, but it is incremental as it builds on existing methods.
The paper tackled the problem of imposing monotonicity in neural networks by generalizing theoretical results to show that MLPs with non-negative weight constraints and activations saturating on alternating sides are universal approximators for monotonic functions, and it proposed an alternative formulation that eliminates weight reparameterization, improving training stability and achieving competitive performance in experiments.
Conventional techniques for imposing monotonicity in MLPs by construction involve the use of non-negative weight constraints and bounded activation functions, which pose well-known optimization challenges. In this work, we generalize previous theoretical results, showing that MLPs with non-negative weight constraint and activations that saturate on alternating sides are universal approximators for monotonic functions. Additionally, we show an equivalence between the saturation side in the activations and the sign of the weight constraint. This connection allows us to prove that MLPs with convex monotone activations and non-positive constrained weights also qualify as universal approximators, in contrast to their non-negative constrained counterparts. Our results provide theoretical grounding to the empirical effectiveness observed in previous works while leading to possible architectural simplification. Moreover, to further alleviate the optimization difficulties, we propose an alternative formulation that allows the network to adjust its activations according to the sign of the weights. This eliminates the requirement for weight reparameterization, easing initialization and improving training stability. Experimental evaluation reinforces the validity of the theoretical results, showing that our novel approach compares favourably to traditional monotonic architectures.