On Pitfalls of Measuring Occlusion Robustness through Data Distortion
This addresses a methodological issue in computer vision evaluation, particularly for occlusion robustness, which is incremental as it refines existing practices.
The paper tackles the problem of biased occlusion robustness evaluation due to artefacts from data distortion, showing that ignoring these artefacts leads to unfair results, and proposes iOcclusion as a fairer alternative for unknown occluders.
Over the past years, the crucial role of data has largely been shadowed by the field's focus on architectures and training procedures. We often cause changes to the data without being aware of their wider implications. In this paper we show that distorting images without accounting for the artefacts introduced leads to biased results when establishing occlusion robustness. To ensure models behave as expected in real-world scenarios, we need to rule out the impact added artefacts have on evaluation. We propose a new approach, iOcclusion, as a fairer alternative for applications where the possible occluders are unknown.