Subhadeep Karan

Marina Blanton, Ah Reum Kang, Subhadeep Karan et al.

Objective: To enable privacy-preserving learning of high quality generative and discriminative machine learning models from distributed electronic health records. Methods and Results: We describe general and scalable strategy to build machine learning models in a provably privacy-preserving way. Compared to the standard approaches using, e.g., differential privacy, our method does not require alteration of the input biomedical data, works with completely or partially distributed datasets, and is resilient as long as the majority of the sites participating in data processing are trusted to not collude. We show how the proposed strategy can be applied on distributed medical records to solve the variables assignment problem, the key task in exact feature selection and Bayesian networks learning. Conclusions: Our proposed architecture can be used by health care organizations, spanning providers, insurers, researchers and computational service providers, to build robust and high quality predictive models in cases where distributed data has to be combined without being disclosed, altered or otherwise compromised.

Subhadeep Karan, Matthew Eichhorn, Blake Hurlburt et al.

We propose scalable methods to execute counting queries in machine learning applications. To achieve memory and computational efficiency, we abstract counting queries and their context such that the counts can be aggregated as a stream. We demonstrate performance and scalability of the resulting approach on random queries, and through extensive experimentation using Bayesian networks learning and association rule mining. Our methods significantly outperform commonly used ADtrees and hash tables, and are practical alternatives for processing large-scale data.

Subhadeep Karan, Jaroslaw Zola

Bayesian networks are probabilistic graphical models often used in big data analytics. The problem of exact structure learning is to find a network structure that is optimal under certain scoring criteria. The problem is known to be NP-hard and the existing methods are both computationally and memory intensive. In this paper, we introduce a new approach for exact structure learning. Our strategy is to leverage relationship between a partial network structure and the remaining variables to constraint the number of ways in which the partial network can be optimally extended. Via experimental results, we show that the method provides up to three times improvement in runtime, and orders of magnitude reduction in memory consumption over the current best algorithms.