Evaluating SAP Joule for Code Generation
This provides a comparative benchmark for SAP Joule's performance in code generation, which is incremental as it assesses an existing model against others.
This paper evaluated SAP Joule's code generation capabilities on the HumanEval-X Javascript benchmark, achieving a strict accuracy of 80.49% and ranking fifth among 29 models.
SAP has released its own proprietary generative model SAP Joule, intended for various generative tasks, including serving as a code assistant for software engineers. While Joule is yet not focused on SAP-specific ABAP code generation, it can be used for other common languages, including Javascript. This paper compares SAP Joules Javascript coding capabilities against a total of 29 other models using the HumanEval-X Javascript benchmark. SAP Joule achieves a strict accuracy of 80.49% as the fifth best model in our evaluation. To the best of our knowledge, this is the first comparative evaluation of SAP Joule code generation capabilities.