Lipschitz standardization for multivariate learning
This addresses the issue of imbalanced learning in multivariate probabilistic models, particularly for datasets with mixed continuous and discrete variables, though it is incremental as it builds on multitask learning ideas for data preprocessing.
The paper tackles the problem of multivariate probabilistic learning where models often fit only a subset of variables, overlooking others, by proposing Lipschitz standardization as a data preprocessing method to balance local Lipschitz smoothness across variables, resulting in more accurate models than existing techniques on real-world datasets.
Probabilistic learning is increasingly being tackled as an optimization problem, with gradient-based approaches as predominant methods. When modelling multivariate likelihoods, a usual but undesirable outcome is that the learned model fits only a subset of the observed variables, overlooking the rest. In this work, we study this problem through the lens of multitask learning (MTL), where similar effects have been broadly studied. While MTL solutions do not directly apply in the probabilistic setting (as they cannot handle the likelihood constraints) we show that similar ideas may be leveraged during data preprocessing. First, we show that data standardization often helps under common continuous likelihoods, but it is not enough in the general case, specially under mixed continuous and discrete likelihood models. In order for balance multivariate learning, we then propose a novel data preprocessing, Lipschitz standardization, which balances the local Lipschitz smoothness across variables. Our experiments on real-world datasets show that Lipschitz standardization leads to more accurate multivariate models than the ones learned using existing data preprocessing techniques. The models and datasets employed in the experiments can be found in https://github.com/adrianjav/lipschitz-standardization.