RANalyzer: Automated Continuous RAN Software Evaluation and Regression Analysis
For O-RAN developers and operators, RANalyzer addresses the challenge of attributing performance variations to software changes in stochastic wireless environments, replacing manual troubleshooting with automated regression analysis.
RANalyzer automates the evaluation of software updates in O-RAN systems by combining LLM-assisted semantic extraction with residuals analysis to identify performance degradations beyond those caused by wireless channel conditions. Over 8,600 tests across 69 releases, it successfully correlates code changes with performance impacts, enabling integration into CI/CD/CT pipelines.
Software-driven O-RAN architectures enable rapid innovation through frequent, independent updates to virtualized components. However, attributing performance variations to specific software changes is challenging due to the stochastic nature of wireless systems, where channel conditions, interference, and hardware variability confound analysis. Traditional threshold-based monitoring and manual troubleshooting do not scale with modern software evolution. This paper presents RANalyzer, an automated test analysis framework that quantifies the performance impact of software updates beyond what can be explained by wireless channel conditions. RANalyzer combines LLM-assisted semantic extraction with residuals analysis. The first categorizes code changes by affected protocol layers and functional components, while the second provides insights on the effect of load, channel, or code changes on the test performance. We contribute an extensive dataset collected over more than two years of continuous over-the-air testing on an experimental O-RAN testbed, comprising over 8,600 automated tests across 69 releases of the OAI stack. By modeling expected performance and interpreting deviations as software-induced effects, we identify degraded instances attributable to code changes and correlate them with specific change categories. The framework can be integrated into CI/CD/CT pipelines for automated, continuous evaluation of software updates at scale.