On Enhancing Root Cause Analysis with SQL Summaries for Failures in Database Workload Replays at SAP HANA
This work addresses the challenge of improving root cause analysis for database failures at SAP HANA, though it is incremental as it builds on existing machine learning frameworks.
The paper tackles the problem of false positive errors in database workload replay regression testing by using a large language model to generate SQL summaries as features for failure classification, resulting in a 4.77% improvement in F1-Macro score.
Capturing the workload of a database and replaying this workload for a new version of the database can be an effective approach for regression testing. However, false positive errors caused by many factors such as data privacy limitations, time dependency or non-determinism in multi-threaded environment can negatively impact the effectiveness. Therefore, we employ a machine learning based framework to automate the root cause analysis of failures found during replays. However, handling unseen novel issues not found in the training data is one general challenge of machine learning approaches with respect to generalizability of the learned model. We describe how we continue to address this challenge for more robust long-term solutions. From our experience, retraining with new failures is inadequate due to features overlapping across distinct root causes. Hence, we leverage a large language model (LLM) to analyze failed SQL statements and extract concise failure summaries as an additional feature to enhance the classification process. Our experiments show the F1-Macro score improved by 4.77% for our data. We consider our approach beneficial for providing end users with additional information to gain more insights into the found issues and to improve the assessment of the replay results.