Some thoughts about benchmarks for NMR
This addresses the lack of standardized benchmarks for NMR systems, which is an incremental step to facilitate progress in this domain.
The paper discusses the need for a benchmark repository in the NMR community to advance system design, drawing lessons from other AI areas.
The NMR community would like to build a repository of benchmarks to push forward the design of systems implementing NMR as it has been the case for many other areas in AI. There are a number of lessons which can be learned from the experience of other communi- ties. Here are a few thoughts about the requirements and choices to make before building such a repository.