Efficient and Scalable Structure Learning for Bayesian Networks: Algorithms and Applications
This work provides a highly efficient and scalable Bayesian network structure learning algorithm, which is crucial for large-scale real-world applications within Alibaba Group and potentially other domains requiring fast anomaly detection and root cause analysis.
The paper introduces LEAST, a new algorithm for learning Bayesian network structures that addresses limitations in efficiency and scalability. LEAST formulates structure learning as a continuous constrained optimization problem with a novel differentiable acyclicity constraint, achieving 1 to 2 orders of magnitude faster performance than state-of-the-art methods while maintaining comparable accuracy.
Structure Learning for Bayesian network (BN) is an important problem with extensive research. It plays central roles in a wide variety of applications in Alibaba Group. However, existing structure learning algorithms suffer from considerable limitations in real world applications due to their low efficiency and poor scalability. To resolve this, we propose a new structure learning algorithm LEAST, which comprehensively fulfills our business requirements as it attains high accuracy, efficiency and scalability at the same time. The core idea of LEAST is to formulate the structure learning into a continuous constrained optimization problem, with a novel differentiable constraint function measuring the acyclicity of the resulting graph. Unlike with existing work, our constraint function is built on the spectral radius of the graph and could be evaluated in near linear time w.r.t. the graph node size. Based on it, LEAST can be efficiently implemented with low storage overhead. According to our benchmark evaluation, LEAST runs 1 to 2 orders of magnitude faster than state of the art method with comparable accuracy, and it is able to scale on BNs with up to hundreds of thousands of variables. In our production environment, LEAST is deployed and serves for more than 20 applications with thousands of executions per day. We describe a concrete scenario in a ticket booking service in Alibaba, where LEAST is applied to build a near real-time automatic anomaly detection and root error cause analysis system. We also show that LEAST unlocks the possibility of applying BN structure learning in new areas, such as large-scale gene expression data analysis and explainable recommendation system.