Trust Calibration in IDEs: Paving the Way for Widespread Adoption of AI Refactoring
This work tackles the problem of trust and safety in AI refactoring for software developers, but it is incremental as it builds on existing human factors research without presenting new results.
The paper addresses the challenge of adopting AI-assisted refactoring in software development by highlighting risks like breaking changes and security vulnerabilities, proposing to integrate LLMs in IDEs with safeguards and trust-building interactions to enable widespread use.
In the software industry, the drive to add new features often overshadows the need to improve existing code. Large Language Models (LLMs) offer a new approach to improving codebases at an unprecedented scale through AI-assisted refactoring. However, LLMs come with inherent risks such as braking changes and the introduction of security vulnerabilities. We advocate for encapsulating the interaction with the models in IDEs and validating refactoring attempts using trustworthy safeguards. However, equally important for the uptake of AI refactoring is research on trust development. In this position paper, we position our future work based on established models from research on human factors in automation. We outline action research within CodeScene on development of 1) novel LLM safeguards and 2) user interaction that conveys an appropriate level of trust. The industry collaboration enables large-scale repository analysis and A/B testing to continuously guide the design of our research interventions.