JupOtter: Cell-Level Bug Detection in Jupyter Notebooks
For data scientists and developers using Jupyter Notebooks, JupOtter addresses the growing problem of buggy notebooks with a specialized detection method.
JupOtter introduces a cell-level bug detection system for Jupyter Notebooks, achieving F1 scores that surpass static analyzers and LLMs in two of three datasets, using a novel tokenization and cell-level prediction technique.
Jupyter Notebooks are an increasingly popular coding environment used across many domains, especially in Python-based data science and scientific computing. Originally used for prototyping and interactive exploration, notebooks are increasingly used to develop more complex programs, leading to a rapid rise in buggy notebooks on platforms like GitHub. To address this trend, we present JupOtter, a bug detection system designed specifically for Jupyter Notebooks. JupOtter features three novel contributions: (1) a notebook-specific tokenization strategy that preserves cell structure, (2) a cell-level bug prediction technique, and (3) a new labeled dataset, OtterDataset, containing over 21,000 notebooks annotated for fine-grained cell-level bug detection. JupOtter achieves cell-level bug detection F1 scores that surpass static analyzers and large language models in two out of three evaluation datasets.