CodeBPE: Investigating Subtokenization Options for Large Language Model Pretraining on Source Code
This work addresses subtokenization efficiency for developers and researchers using large language models on source code, but it is incremental as it builds on existing methods.
The paper tackled the problem of subtokenization for large language models in source code by investigating different options to improve efficiency and performance, resulting in a 17% reduction in average length without performance drop and potential quality improvements of 0.5-2%.
Recent works have widely adopted large language model pretraining for source code, suggested source code-specific pretraining objectives and investigated the applicability of various Transformer-based language model architectures for source code. This work investigates another important aspect of such models, namely the effect of different subtokenization options, and aims at identifying most effective and length-efficient subtokenizations, taking into account code specifics. We propose subtokenziation that reduces average length by 17% without downstream performance drop, and show that a carefully chosen subtokenization may improve quality by 0.5-2%, possibly with some length increase.