Proba-V-ref: Repurposing the Proba-V challenge for reference-aware super resolution
This work addresses a flaw in a benchmark dataset for satellite image super-resolution, which is important for researchers and practitioners in remote sensing and computer vision, though it is incremental as it modifies an existing dataset rather than introducing a new method.
The authors identified that the PROBA-V Super-Resolution challenge inadvertently ranks methods based on heuristics for selecting reference images rather than pure Multi-Image Super Resolution (MISR) performance, and they demonstrated this by improving winners' results using a simple heuristic to choose a different reference image.
The PROBA-V Super-Resolution challenge distributes real low-resolution image series and corresponding high-resolution targets to advance research on Multi-Image Super Resolution (MISR) for satellite images. However, in the PROBA-V dataset the low-resolution image corresponding to the high-resolution target is not identified. We argue that in doing so, the challenge ranks the proposed methods not only by their MISR performance, but mainly by the heuristics used to guess which image in the series is the most similar to the high-resolution target. We demonstrate this by improving the performance obtained by the two winners of the challenge only by using a different reference image, which we compute following a simple heuristic. Based on this, we propose PROBA-V-REF a variant of the PROBA-V dataset, in which the reference image in the low-resolution series is provided, and show that the ranking between the methods changes in this setting. This is relevant to many practical use cases of MISR where the goal is to super-resolve a specific image of the series, i.e. the reference is known. The proposed PROBA-V-REF should better reflect the performance of the different methods for this reference-aware MISR problem.