TypyBench: Evaluating LLM Type Inference for Untyped Python Repositories
This work addresses the problem of assessing LLM capabilities for type inference in software engineering, providing a new benchmark for researchers, but it is incremental as it focuses on evaluation rather than proposing a new inference method.
The paper tackles the challenge of evaluating large language models (LLMs) for type inference in untyped Python code by introducing TypyBench, a benchmark with novel metrics, and finds that LLMs achieve decent type similarity scores but struggle with complex nested types and consistency errors.
Type inference for dynamic languages like Python is a persistent challenge in software engineering. While large language models (LLMs) have shown promise in code understanding, their type inference capabilities remain underexplored. We introduce TypyBench, a benchmark designed to evaluate LLMs' type inference across entire Python repositories. TypyBench features two novel metrics: TypeSim, which captures nuanced semantic relationships between predicted and ground truth types, and TypeCheck, which assesses type consistency across codebases. Our evaluation of various LLMs on a curated dataset of 50 high-quality Python repositories reveals that, although LLMs achieve decent TypeSim scores, they struggle with complex nested types and exhibit significant type consistency errors. These findings suggest that future research should shift focus from improving type similarity to addressing repository-level consistency. TypyBench provides a foundation for this new direction, offering insights into model performance across different type complexities and usage contexts. Our code and data are available at https://github.com/typybench/typybench.