Multi-Observation Elicitation
This addresses a foundational problem in machine learning by potentially improving efficiency in property elicitation and empirical risk minimization, though it appears incremental as it builds on existing elicitation concepts.
The paper tackles the problem of prediction accuracy by introducing multi-observation loss functions, which use multiple data points simultaneously, and finds that these can drastically reduce the dimensionality of hypotheses, requiring fewer reports in elicitation and enabling algorithms on smaller-dimensional spaces.
We study loss functions that measure the accuracy of a prediction based on multiple data points simultaneously. To our knowledge, such loss functions have not been studied before in the area of property elicitation or in machine learning more broadly. As compared to traditional loss functions that take only a single data point, these multi-observation loss functions can in some cases drastically reduce the dimensionality of the hypothesis required. In elicitation, this corresponds to requiring many fewer reports; in empirical risk minimization, it corresponds to algorithms on a hypothesis space of much smaller dimension. We explore some examples of the tradeoff between dimensionality and number of observations, give some geometric characterizations and intuition for relating loss functions and the properties that they elicit, and discuss some implications for both elicitation and machine-learning contexts.