Petar Momcilovic

Jiachen Xi, Alfredo Garcia, Petar Momcilovic

Regularized Markov Decision Processes serve as models of sequential decision making under uncertainty wherein the decision maker has limited information processing capacity and/or aversion to model ambiguity. With functional approximation, the convergence properties of learning algorithms for regularized MDPs (e.g. soft Q-learning) are not well understood because the composition of the regularized Bellman operator and a projection onto the span of basis vectors is not a contraction with respect to any norm. In this paper, we consider a bi-level optimization formulation of regularized Q-learning with linear functional approximation. The {\em lower} level optimization problem aims to identify a value function approximation that satisfies Bellman's recursive optimality condition and the {\em upper} level aims to find the projection onto the span of basis vectors. This formulation motivates a single-loop algorithm with finite time convergence guarantees. The algorithm operates on two time-scales: updates to the projection of state-action values are `slow' in that they are implemented with a step size that is smaller than the one used for `faster' updates of approximate solutions to Bellman's recursive optimality equation. We show that, under certain assumptions, the proposed algorithm converges to a stationary point in the presence of Markovian noise. In addition, we provide a performance guarantee for the policies derived from the proposed algorithm.

Anton Kocheturov, Petar Momcilovic, Azra Bihorac et al.

Temporal Pattern Mining (TPM) is the problem of mining predictive complex temporal patterns from multivariate time series in a supervised setting. We develop a new method called the Fast Temporal Pattern Mining with Extended Vertical Lists. This method utilizes an extension of the Apriori property which requires a more complex pattern to appear within records only at places where all of its subpatterns are detected as well. The approach is based on a novel data structure called the Extended Vertical List that tracks positions of the first state of the pattern inside records. Extensive computational results indicate that the new method performs significantly faster than the previous version of the algorithm for TMP. However, the speed-up comes at the expense of memory usage.

Meghan Brennan, Sahil Puri, Tezcan Ozrazgat-Baslanti et al.

Background: Major postoperative complications are associated with increased short and long-term mortality, increased healthcare cost, and adverse long-term consequences. The large amount of data contained in the electronic health record (EHR) creates barriers for physicians to recognize patients most at risk. We hypothesize, if presented in an optimal format, information from data-driven predictive risk algorithms for postoperative complications can improve physician risk assessment. Methods: Prospective, non-randomized, interventional pilot study of twenty perioperative physicians at a quarterly academic medical center. Using 150 clinical cases we compared physicians' risk assessment before and after interaction with MySurgeryRisk, a validated machine-learning algorithm predicting preoperative risk for six major postoperative complications using EHR data. Results: The area under the curve (AUC) of MySurgeryRisk algorithm ranged between 0.73 and 0.85 and was significantly higher than physicians' risk assessments (AUC between 0.47 and 0.69) for all postoperative complications except cardiovascular complications. The AUC for repeated physician's risk assessment improved by 2% to 5% for all complications with the exception of thirty-day mortality. Physicians' risk assessment for acute kidney injury and intensive care unit admission longer than 48 hours significantly improved after knowledge exchange, resulting in net reclassification improvement of 12.4% and 16%, respectively. Conclusions: The validated MySurgeryRisk algorithm predicted postoperative complications with equal or higher accuracy than pilot cohort of physicians using available clinical preoperative data. The interaction with algorithm significantly improved physicians' risk assessment.