Toward a Characterization of Loss Functions for Distribution Learning
This work addresses a foundational gap in distribution learning theory, potentially enabling more flexible loss function selection for researchers and practitioners, though it is incremental as it builds on existing axiomatic approaches.
The paper tackles the lack of diversity in loss functions for distribution learning by proposing axiomatic criteria for evaluating them, finding that no loss satisfies all criteria without restricting to calibrated distributions, but after this restriction, log loss and others meet the criteria, enabling broader application-specific choices.
In this work we study loss functions for learning and evaluating probability distributions over large discrete domains. Unlike classification or regression where a wide variety of loss functions are used, in the distribution learning and density estimation literature, very few losses outside the dominant $log\ loss$ are applied. We aim to understand this fact, taking an axiomatic approach to the design of loss functions for learning distributions. We start by proposing a set of desirable criteria that any good loss function should satisfy. Intuitively, these criteria require that the loss function faithfully evaluates a candidate distribution, both in expectation and when estimated on a few samples. Interestingly, we observe that \emph{no loss function} possesses all of these criteria. However, one can circumvent this issue by introducing a natural restriction on the set of candidate distributions. Specifically, we require that candidates are $calibrated$ with respect to the target distribution, i.e., they may contain less information than the target but otherwise do not significantly distort the truth. We show that, after restricting to this set of distributions, the log loss, along with a large variety of other losses satisfy the desired criteria. These results pave the way for future investigations of distribution learning that look beyond the log loss, choosing a loss function based on application or domain need.