Applying the Chinese Wall Reverse Engineering Technique to Large Language Model Code Editing
This addresses the issue of copyright concerns in code LLMs for developers and researchers, though it is incremental as it builds on existing reverse engineering methods.
The paper tackles the problem of improving the utility of ethically aligned but weaker code LLMs by applying the Chinese Wall reverse engineering technique, which uses a high-quality model to generate instructions for a weaker model, resulting in performance improvements of over 66% for Comma v0.1 1T and roughly 20% for Starcoder2 Instruct on the CanItEdit benchmark.
Large language models for code (Code LLM) are increasingly utilized in programming environments. Despite their utility, the training datasets for top LLM remain undisclosed, raising concerns about potential copyright violations. Some models, such as Pleias and Comma put emphasis on data curation and licenses, however, with limited training data these models are not competitive and only serve as proof of concepts. To improve the utility of these models, we propose an application of the "Chinese Wall" technique, inspired by the reverse engineering technique of the same name -- a high quality model is used to generate detailed instructions for a weaker model. By doing so, a weaker but ethically aligned model may be used to perform complicated tasks that, otherwise, can only be completed by more powerful models. In our evaluation, we've found that this technique improves Comma v0.1 1T's performance in CanItEdit benchmark by over 66%, and Starcoder2 Instruct by roughly 20% compared to when running the same model on the benchmark alone. The practical application of this technique today, however, may be limited due to the lack of models trained on public domain content without copyright restrictions.