Towards Reliable Evaluation of Road Network Reconstructions
This work addresses the challenge of reliably evaluating road network reconstructions for researchers and practitioners in fields like mapping and autonomous driving, representing an incremental improvement in evaluation methodology.
The paper tackled the problem of inconsistent performance measures for road network reconstruction algorithms by identifying design flaws in existing metrics and proposing three new metrics that demonstrate far greater consistency across different evaluation approaches.
Existing performance measures rank delineation algorithms inconsistently, which makes it difficult to decide which one is best in any given situation. We show that these inconsistencies stem from design flaws that make the metrics insensitive to whole classes of errors. To provide more reliable evaluation, we design three new metrics that are far more consistent even though they use very different approaches to comparing ground-truth and reconstructed road networks. We use both synthetic and real data to demonstrate this and advocate the use of these corrected metrics as a tool to gauge future progress.