Mary Monroe

Mary Monroe, Anish Thilagar, Melody Hsu et al.

In forecasting competitions, the traditional mechanism scores the predictions of each contestant against the outcome of each event, and the contestant with the highest total score wins. While it is well-known that this traditional mechanism can suffer from incentive issues, it is folklore that contestants will still be roughly truthful as the number of events grows. Yet thus far the literature lacks a formal analysis of this traditional mechanism. This paper gives the first such analysis. We first demonstrate that the ''long-run truthfulness'' folklore is false: even for arbitrary numbers of events, the best forecaster can have an incentive to hedge, reporting more moderate beliefs to increase their win probability. On the positive side, however, we show that two contestants will be approximately truthful when they have sufficient uncertainty over the relative quality of their opponent and the outcomes of the events, a case which may arise in practice.

Rafael Frongillo, Ian Kash, Mary Monroe

Peer prediction mechanisms are typically proposed and analyzed under the assumption that the report and signal spaces are identical. In practice, however, agents often observe richer information which they then map to a coarser report space. Motivated by this discrepancy between theory and practice, we initiate the study of peer prediction mechanisms with signal spaces that are richer than the report space. We begin by formalizing a model with real-valued signals and binary reports. In this setting, it is natural to study symmetric threshold strategies, where agents map their signals to binary reports according to a single real-valued threshold. For several well-known binary-report peer prediction mechanisms, we show that most equilibria under the original assumption of binary signals are no longer equilibria in our model. Furthermore, dynamic analysis proves that some of the remaining thresholds are unstable. These results extend beyond real-valued signals and binary reports to settings where the signal space is finer-grained than the report space. While the results above suggest important limitations for the deployment of existing peer prediction mechanisms in practice, we also use them to develop a new, more robust mechanism. This mechanism generates a larger number of stable threshold equilibria under our model, thus allowing the designer more flexibility in choosing how agents map their signals to reports.

Rafael Frongillo, Mary Monroe, Eric Neyman et al.

We study the problem of robust forecast aggregation: combining expert forecasts with provable accuracy guarantees compared to the best possible aggregation of the underlying information. Prior work shows strong impossibility results, e.g. that even under natural assumptions, no aggregation of the experts' individual forecasts can outperform simply following a random expert (Neyman and Roughgarden, 2022). In this paper, we introduce a more general framework that allows the principal to elicit richer information from experts through structured queries. Our framework ensures that experts will truthfully report their underlying beliefs, and also enables us to define notions of complexity over the difficulty of asking these queries. Under a general model of independent but overlapping expert signals, we show that optimal aggregation is achievable in the worst case with each complexity measure bounded above by the number of agents $n$. We further establish tight tradeoffs between accuracy and query complexity: aggregation error decreases linearly with the number of queries, and vanishes when the "order of reasoning" and number of agents relevant to a query is $ω(\sqrt{n})$. These results demonstrate that modest extensions to the space of expert queries dramatically strengthen the power of robust forecast aggregation. We therefore expect that our new query framework will open up a fruitful line of research in this area.