LLM-based vs. Search-based Merge Conflict Resolution: An Empirical Study of Competing Paradigms
For software engineering practitioners and tool developers, this study reveals context-dependent trade-offs between LLM and SBSE paradigms, motivating hybrid approaches.
This paper compares LLM-based (MergeGen) and search-based (SBCR) merge conflict resolution tools on thousands of real-world conflicts from four languages. The LLM approach excels on imbalanced conflicts but struggles with non-English content and large inputs, while the SBSE approach generalizes better across datasets and performs best on balanced conflicts.
Context: The resolution of software merge conflicts is being reshaped by two competing paradigms: generative approaches based on Large Language Models (LLMs) and optimization approaches from Search-Based Software Engineering (SBSE). While tools from both paradigms have shown promise, their relative strengths, weaknesses, and trade-offs are not yet well understood. Objective: This paper presents the first in-depth empirical study directly comparing these paradigms to identify their capabilities and limitations in real-world scenarios. Method: We evaluated MergeGen, a state-of-the-art LLM-based tool, against SBCR, a novel SBSE approach employing a Random Restart Hill Climbing (RRHC) algorithm. The comparison used thousands of real-world conflicts from open-source projects written in Java, C#, JavaScript, and TypeScript. Results: Our findings reveal fundamental trade-offs. The LLM paradigm excels at resolving conflicts with imbalanced content by leveraging learned patterns. However, it struggles with non-English content and large inputs, which can lead to truncated or empty resolutions. Conversely, the SBSE paradigm demonstrates superior generalization across datasets and performs best on balanced conflicts, highlighting its potential as a robust, data-independent alternative. Conclusions: Neither paradigm is a silver bullet. Our findings highlight context-dependent strengths and motivate the development of hybrid systems that combine the complementary capabilities of LLM and SBSE approaches to create more robust and reliable merge conflict resolution tools.