DDTS: A Practical System Testing Framework for Scientific Software
This addresses testing challenges for scientific software developers, but it is incremental as it builds on existing testing concepts.
The paper tackles the problem of inadequate testing in scientific software by introducing DDTS, a framework that simplifies the adoption and scaling of rigorous testing, and it is currently used for automated regression and pre-commit testing in several projects.
Many scientific-software projects test their codes inadequately, or not at all. Despite its well-known benefits, adopting routine testing is often not easy. Development teams may have doubts about establishing effective test procedures, writing test software, or handling the ever-growing complexity of test cases. They may need to run (and test) on restrictive HPC platforms. They almost certainly face time and budget pressures that can keep testing languishing near the bottom of their to-do lists. This paper presents DDTS, a framework for building test suite applications, designed to fit scientific-software projects' requirements. DDTS aims to simplify introduction of rigorous testing, and to ease growing pains as needs mature. It decomposes the testing problem into practical, intuitive phases; makes configuration and extension easy; is portable and suitable to HPC platforms; and exploits parallelism. DDTS is currently used for automated regression and developer pre-commit testing for several scientific-software projects with disparate testing requirements.