Testing with Jupyter notebooks: NoteBook VALidation (nbval) plug-in for pytest
This tool addresses the need for automated testing in notebook-based documentation and tutorials, as well as for unit and integration tests in libraries, though it is incremental as it builds on existing pytest infrastructure.
The authors tackled the problem of validating Jupyter notebooks by developing nbval, a pytest plugin that executes notebook cells and compares outputs to saved ones, reporting deviations as test failures.
The Notebook validation tool nbval allows to load and execute Python code from a Jupyter notebook file. While computing outputs from the cells in the notebook, these outputs are compared with the outputs saved in the notebook file, treating each cell as a test. Deviations are reported as test failures, with various configuration options available to control the behaviour. Application use cases include the validation of notebook-based documentation, tutorials and textbooks, as well as the use of notebooks as additional unit, integration and system tests for the libraries that are used in the notebook. Nbval is implemented as a plugin for the pytest testing software.