Aggregating Correlated Estimations with (Almost) no Training
This work addresses a specific issue in decision-making systems for practitioners, but it is incremental as it builds on existing aggregation methods.
The paper tackles the problem of aggregating correlated estimation errors from multiple algorithms by proposing new aggregation rules that account for correlations, showing that maximum likelihood aggregation is best when correlations are known, while Embedded Voting is recommended with limited training data.
Many decision problems cannot be solved exactly and use several estimation algorithms that assign scores to the different available options. The estimation errors can have various correlations, from low (e.g. between two very different approaches) to high (e.g. when using a given algorithm with different hyperparameters). Most aggregation rules would suffer from this diversity of correlations. In this article, we propose different aggregation rules that take correlations into account, and we compare them to naive rules in various experiments based on synthetic data. Our results show that when sufficient information is known about the correlations between errors, a maximum likelihood aggregation should be preferred. Otherwise, typically with limited training data, we recommend a method that we call Embedded Voting (EV).